2024-03-23T20:40:13.150 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T20:40:13.150 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668 --owner scheduled_yuriw@teuthology --description rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus} --verbose -- /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/orig.config.yaml 2024-03-23T20:40:13.251 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T20:40:13.337 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668 branch: wip-yuri11-testing-2024-03-21-0851-reef description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus} email: yweinste@redhat.com first_in_suite: false job_id: '7617668' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon name: yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri11-testing-2024-03-21-0851-reef ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 82316b04821a6c2c1d458a881920454bbadb8837 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 82316b04821a6c2c1d458a881920454bbadb8837 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri11-testing-2024-03-21-0851-reef sha1: 82316b04821a6c2c1d458a881920454bbadb8837 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 6304 sha1: 82316b04821a6c2c1d458a881920454bbadb8837 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuri11-testing-2024-03-21-0851-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 82316b04821a6c2c1d458a881920454bbadb8837 targets: smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1jcoioqpma6131vrwi23FPocv1RZFh/qwKKJR0unECZSCSz46OXgHJIhVax58gSn+DqTttcf+hwf4JLjyGxfs= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP3lLbZ167RBEDXN5OmvB8wahaNtp28lVj7Ctgps/XZan0ZhVsbikIRpq37YPGsjrb4IZAdK/KxIOad73zDNz7Q= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_journal.sh env: CEPH_ARGS: '' RBD_MIRROR_MIN_COMPAT_CLIENT: octopus RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_13:19:01 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-23T20:40:13.337 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa; will attempt to use it 2024-03-23T20:40:13.337 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa/tasks 2024-03-23T20:40:13.338 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T20:40:13.340 INFO:teuthology.task.internal:Checking packages... 2024-03-23T20:40:13.364 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '82316b04821a6c2c1d458a881920454bbadb8837' 2024-03-23T20:40:13.364 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T20:40:13.364 INFO:teuthology.packaging:ref: None 2024-03-23T20:40:13.364 INFO:teuthology.packaging:tag: None 2024-03-23T20:40:13.364 INFO:teuthology.packaging:branch: wip-yuri11-testing-2024-03-21-0851-reef 2024-03-23T20:40:13.364 INFO:teuthology.packaging:sha1: 82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T20:40:13.364 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri11-testing-2024-03-21-0851-reef 2024-03-23T20:40:13.583 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-954-g82316b04-1jammy 2024-03-23T20:40:13.587 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T20:40:13.593 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T20:40:13.593 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T20:40:13.605 INFO:teuthology.task.internal:Saving configuration 2024-03-23T20:40:13.623 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T20:40:13.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T20:40:13.660 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 20:34:16.383409', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH3RuWGFk/3S8Iq3orMUCLvEf9RiCEqSZJiaMAFY2uZj'} 2024-03-23T20:40:13.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 20:34:16.381506', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEdybu2xQk6kancXQriP8KcGXhb9M8gDjF0A2zCKMT++'} 2024-03-23T20:40:13.681 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T20:40:13.692 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-03-23T20:40:13.692 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-03-23T20:40:13.692 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T20:40:13.771 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5ac1e14430>, signals=[15]) 2024-03-23T20:40:13.772 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T20:40:13.778 INFO:teuthology.task.internal:Opening connections... 2024-03-23T20:40:13.778 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T20:40:13.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:40:13.859 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T20:40:13.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:40:13.940 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T20:40:13.949 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-03-23T20:40:13.954 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-03-23T20:40:13.954 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="Ubuntu" 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="22.04" 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_CODENAME=jammy 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:ID=ubuntu 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE=debian 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-23T20:40:14.003 INFO:teuthology.orchestra.run.smithi117.stdout:UBUNTU_CODENAME=jammy 2024-03-23T20:40:14.004 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-03-23T20:40:14.025 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-03-23T20:40:14.030 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-03-23T20:40:14.030 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="22.04" 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=jammy 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-23T20:40:14.079 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-23T20:40:14.080 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-23T20:40:14.080 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-23T20:40:14.080 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=jammy 2024-03-23T20:40:14.080 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-03-23T20:40:14.108 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T20:40:14.122 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T20:40:14.172 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T20:40:14.173 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-03-23T20:40:14.175 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-03-23T20:40:14.180 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T20:40:14.241 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T20:40:14.241 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-03-23T20:40:14.244 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-03-23T20:40:14.251 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T20:40:14.306 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T20:40:14.325 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-23T20:40:14.325 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-23T20:40:14.325 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T20:40:14.326 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T20:40:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:40:14.330 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-23T20:40:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:40:14.331 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-23T20:40:14.375 INFO:teuthology.orchestra.run.smithi117.stdout:5.15.0-56-generic 2024-03-23T20:40:14.375 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.15.0-56-generic 2024-03-23T20:40:14.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get clean 2024-03-23T20:40:14.380 INFO:teuthology.orchestra.run.smithi204.stdout:5.15.0-56-generic 2024-03-23T20:40:14.380 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.15.0-56-generic 2024-03-23T20:40:14.380 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2024-03-23T20:40:14.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2024-03-23T20:40:14.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2024-03-23T20:40:14.625 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-23T20:40:14.639 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-23T20:40:14.644 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-23T20:40:14.665 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-23T20:40:14.713 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-23T20:40:14.731 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-23T20:40:14.744 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-23T20:40:14.756 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-23T20:40:14.848 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [594 kB] 2024-03-23T20:40:14.872 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [594 kB] 2024-03-23T20:40:14.926 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,502 kB] 2024-03-23T20:40:14.954 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,502 kB] 2024-03-23T20:40:14.963 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [289 kB] 2024-03-23T20:40:14.969 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-23T20:40:14.969 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-23T20:40:14.969 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,619 kB] 2024-03-23T20:40:14.991 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [271 kB] 2024-03-23T20:40:14.994 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-23T20:40:14.995 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [697 kB] 2024-03-23T20:40:15.004 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,059 kB] 2024-03-23T20:40:15.007 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [289 kB] 2024-03-23T20:40:15.018 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-23T20:40:15.018 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-23T20:40:15.020 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,619 kB] 2024-03-23T20:40:15.033 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [240 kB] 2024-03-23T20:40:15.035 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-23T20:40:15.035 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-23T20:40:15.036 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-23T20:40:15.036 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-23T20:40:15.036 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-23T20:40:15.040 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [271 kB] 2024-03-23T20:40:15.047 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-23T20:40:15.047 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [697 kB] 2024-03-23T20:40:15.055 INFO:teuthology.orchestra.run.smithi117.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,059 kB] 2024-03-23T20:40:15.059 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-23T20:40:15.060 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-23T20:40:15.061 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-23T20:40:15.069 INFO:teuthology.orchestra.run.smithi117.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [240 kB] 2024-03-23T20:40:15.079 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-23T20:40:15.079 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-23T20:40:15.080 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-23T20:40:15.080 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-23T20:40:15.080 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-23T20:40:15.091 INFO:teuthology.orchestra.run.smithi117.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-23T20:40:15.092 INFO:teuthology.orchestra.run.smithi117.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-23T20:40:15.092 INFO:teuthology.orchestra.run.smithi117.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-23T20:40:15.093 INFO:teuthology.orchestra.run.smithi117.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-23T20:40:15.093 INFO:teuthology.orchestra.run.smithi117.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-23T20:40:15.093 INFO:teuthology.orchestra.run.smithi117.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-23T20:40:15.094 INFO:teuthology.orchestra.run.smithi117.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-23T20:40:15.094 INFO:teuthology.orchestra.run.smithi117.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-23T20:40:15.094 INFO:teuthology.orchestra.run.smithi117.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-23T20:40:15.116 INFO:teuthology.orchestra.run.smithi117.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-23T20:40:15.117 INFO:teuthology.orchestra.run.smithi117.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-23T20:40:15.117 INFO:teuthology.orchestra.run.smithi117.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-23T20:40:15.134 INFO:teuthology.orchestra.run.smithi117.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-23T20:40:15.153 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,286 kB] 2024-03-23T20:40:15.198 INFO:teuthology.orchestra.run.smithi117.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,286 kB] 2024-03-23T20:40:15.555 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [428 kB] 2024-03-23T20:40:15.573 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [229 kB] 2024-03-23T20:40:15.582 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-23T20:40:15.583 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-23T20:40:15.584 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,591 kB] 2024-03-23T20:40:15.599 INFO:teuthology.orchestra.run.smithi117.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [428 kB] 2024-03-23T20:40:15.617 INFO:teuthology.orchestra.run.smithi117.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [229 kB] 2024-03-23T20:40:15.627 INFO:teuthology.orchestra.run.smithi117.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-23T20:40:15.628 INFO:teuthology.orchestra.run.smithi117.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,591 kB] 2024-03-23T20:40:15.676 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [267 kB] 2024-03-23T20:40:15.687 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-23T20:40:15.687 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-23T20:40:15.717 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-23T20:40:15.719 INFO:teuthology.orchestra.run.smithi117.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-23T20:40:15.720 INFO:teuthology.orchestra.run.smithi117.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [267 kB] 2024-03-23T20:40:15.731 INFO:teuthology.orchestra.run.smithi117.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-23T20:40:15.731 INFO:teuthology.orchestra.run.smithi117.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-23T20:40:15.762 INFO:teuthology.orchestra.run.smithi117.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-23T20:40:15.893 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-23T20:40:15.939 INFO:teuthology.orchestra.run.smithi117.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-23T20:40:16.336 INFO:teuthology.orchestra.run.smithi117.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-23T20:40:16.337 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-23T20:40:16.342 INFO:teuthology.orchestra.run.smithi117.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-23T20:40:16.344 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-23T20:40:16.344 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-23T20:40:16.356 INFO:teuthology.orchestra.run.smithi117.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-23T20:40:16.356 INFO:teuthology.orchestra.run.smithi117.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-23T20:40:16.358 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-23T20:40:16.359 INFO:teuthology.orchestra.run.smithi117.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-23T20:40:16.361 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-23T20:40:20.659 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 12.5 MB in 2s (6,884 kB/s) 2024-03-23T20:40:20.676 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 12.5 MB in 2s (6,993 kB/s) 2024-03-23T20:40:21.999 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-03-23T20:40:22.006 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-03-23T20:40:22.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-23T20:40:22.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-23T20:40:22.075 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-03-23T20:40:22.134 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-03-23T20:40:22.270 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2024-03-23T20:40:22.271 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2024-03-23T20:40:22.313 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2024-03-23T20:40:22.314 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2024-03-23T20:40:22.435 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2024-03-23T20:40:22.435 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-23T20:40:22.436 INFO:teuthology.orchestra.run.smithi117.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-23T20:40:22.436 INFO:teuthology.orchestra.run.smithi117.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-23T20:40:22.436 INFO:teuthology.orchestra.run.smithi117.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-23T20:40:22.436 INFO:teuthology.orchestra.run.smithi117.stdout: python2.7-minimal 2024-03-23T20:40:22.436 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-23T20:40:22.437 INFO:teuthology.orchestra.run.smithi117.stdout:The following additional packages will be installed: 2024-03-23T20:40:22.437 INFO:teuthology.orchestra.run.smithi117.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-23T20:40:22.437 INFO:teuthology.orchestra.run.smithi117.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-23T20:40:22.438 INFO:teuthology.orchestra.run.smithi117.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-23T20:40:22.439 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2024-03-23T20:40:22.439 INFO:teuthology.orchestra.run.smithi117.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-23T20:40:22.439 INFO:teuthology.orchestra.run.smithi117.stdout:Recommended packages: 2024-03-23T20:40:22.439 INFO:teuthology.orchestra.run.smithi117.stdout: thermald 2024-03-23T20:40:22.465 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2024-03-23T20:40:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-23T20:40:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-23T20:40:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-23T20:40:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-23T20:40:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: python2.7-minimal 2024-03-23T20:40:22.466 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-23T20:40:22.467 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2024-03-23T20:40:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-23T20:40:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-23T20:40:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-23T20:40:22.469 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2024-03-23T20:40:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-23T20:40:22.469 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2024-03-23T20:40:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: thermald 2024-03-23T20:40:22.485 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2024-03-23T20:40:22.486 INFO:teuthology.orchestra.run.smithi117.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-23T20:40:22.486 INFO:teuthology.orchestra.run.smithi117.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-23T20:40:22.486 INFO:teuthology.orchestra.run.smithi117.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-23T20:40:22.486 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages will be upgraded: 2024-03-23T20:40:22.487 INFO:teuthology.orchestra.run.smithi117.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-23T20:40:22.519 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2024-03-23T20:40:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-23T20:40:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-23T20:40:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-23T20:40:22.520 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages will be upgraded: 2024-03-23T20:40:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-23T20:40:22.566 INFO:teuthology.orchestra.run.smithi117.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-23T20:40:22.566 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 113 MB of archives. 2024-03-23T20:40:22.566 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-23T20:40:22.567 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-23T20:40:22.578 INFO:teuthology.orchestra.run.smithi204.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-23T20:40:22.578 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 113 MB of archives. 2024-03-23T20:40:22.578 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-23T20:40:22.578 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-23T20:40:23.334 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-23T20:40:23.524 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-23T20:40:23.525 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-23T20:40:23.941 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-23T20:40:25.709 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-23T20:40:25.709 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-23T20:40:25.709 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-23T20:40:25.901 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-23T20:40:25.953 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-23T20:40:26.323 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-23T20:40:26.324 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-23T20:40:26.324 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-23T20:40:26.331 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 113 MB in 3s (32.7 MB/s) 2024-03-23T20:40:26.511 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-23T20:40:26.517 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-23T20:40:26.544 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-23T20:40:26.929 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 113 MB in 4s (28.2 MB/s) 2024-03-23T20:40:27.088 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-23T20:40:28.480 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-23T20:40:28.484 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-23T20:40:28.564 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:29.135 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-23T20:40:29.139 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-23T20:40:29.336 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:30.740 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-23T20:40:30.761 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-23T20:40:30.840 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:31.487 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-23T20:40:31.509 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-23T20:40:31.588 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:31.926 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-23T20:40:31.947 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-23T20:40:31.984 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:32.049 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-23T20:40:32.070 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-23T20:40:32.132 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:38.407 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-23T20:40:38.535 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-23T20:40:38.859 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-23T20:40:38.963 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-23T20:40:39.018 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-23T20:40:39.135 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-23T20:40:39.224 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-23T20:40:39.235 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-23T20:40:39.274 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-23T20:40:39.459 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-23T20:40:39.580 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-23T20:40:39.892 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-23T20:40:39.914 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-23T20:40:39.975 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-23T20:40:44.856 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-23T20:40:44.892 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-23T20:40:44.930 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:45.848 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-23T20:40:45.885 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-23T20:40:45.923 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:48.049 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-23T20:40:48.173 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-23T20:40:48.498 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-23T20:40:48.555 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-23T20:40:48.594 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:48.677 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-23T20:40:48.704 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-23T20:40:48.832 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:49.044 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-23T20:40:49.157 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:49.275 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-23T20:40:49.387 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:50.298 INFO:teuthology.orchestra.run.smithi117.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-23T20:40:50.298 INFO:teuthology.orchestra.run.smithi117.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-23T20:40:50.298 INFO:teuthology.orchestra.run.smithi117.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-23T20:40:50.298 INFO:teuthology.orchestra.run.smithi117.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-23T20:40:50.709 INFO:teuthology.orchestra.run.smithi204.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-23T20:40:50.709 INFO:teuthology.orchestra.run.smithi204.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-23T20:40:50.709 INFO:teuthology.orchestra.run.smithi204.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-23T20:40:50.709 INFO:teuthology.orchestra.run.smithi204.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-23T20:40:50.924 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:50.952 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:52.341 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:52.360 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:53.742 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-23T20:40:53.787 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-23T20:40:53.868 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-23T20:40:53.912 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-23T20:40:53.995 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:54.039 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-23T20:40:54.067 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-23T20:40:54.068 INFO:teuthology.orchestra.run.smithi204.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-23T20:40:54.100 INFO:teuthology.orchestra.run.smithi117.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-23T20:40:54.101 INFO:teuthology.orchestra.run.smithi117.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-23T20:41:07.833 INFO:teuthology.orchestra.run.smithi117.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-23T20:41:07.833 INFO:teuthology.orchestra.run.smithi117.stdout:Sourcing file `/etc/default/grub' 2024-03-23T20:41:07.839 INFO:teuthology.orchestra.run.smithi117.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-23T20:41:07.853 INFO:teuthology.orchestra.run.smithi117.stdout:Generating grub configuration file ... 2024-03-23T20:41:08.007 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-23T20:41:08.008 INFO:teuthology.orchestra.run.smithi204.stdout:Sourcing file `/etc/default/grub' 2024-03-23T20:41:08.023 INFO:teuthology.orchestra.run.smithi204.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-23T20:41:08.045 INFO:teuthology.orchestra.run.smithi204.stdout:Generating grub configuration file ... 2024-03-23T20:41:08.136 INFO:teuthology.orchestra.run.smithi117.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-23T20:41:08.159 INFO:teuthology.orchestra.run.smithi117.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-23T20:41:08.360 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-23T20:41:08.375 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-23T20:41:08.500 INFO:teuthology.orchestra.run.smithi117.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-23T20:41:08.505 INFO:teuthology.orchestra.run.smithi117.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-23T20:41:08.574 INFO:teuthology.orchestra.run.smithi117.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-23T20:41:08.580 INFO:teuthology.orchestra.run.smithi117.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-23T20:41:08.695 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-23T20:41:08.701 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-23T20:41:08.743 INFO:teuthology.orchestra.run.smithi117.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-23T20:41:08.743 INFO:teuthology.orchestra.run.smithi117.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-23T20:41:08.743 INFO:teuthology.orchestra.run.smithi117.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-23T20:41:08.769 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-23T20:41:08.775 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-23T20:41:08.790 INFO:teuthology.orchestra.run.smithi117.stdout:done 2024-03-23T20:41:08.935 INFO:teuthology.orchestra.run.smithi204.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-23T20:41:08.936 INFO:teuthology.orchestra.run.smithi204.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-23T20:41:08.936 INFO:teuthology.orchestra.run.smithi204.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-23T20:41:08.976 INFO:teuthology.orchestra.run.smithi204.stdout:done 2024-03-23T20:41:09.304 DEBUG:teuthology.orchestra.run.smithi117:> dpkg -s linux-image-generic 2024-03-23T20:41:09.365 INFO:teuthology.orchestra.run.smithi117.stdout:Package: linux-image-generic 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Status: install ok installed 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Priority: optional 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Section: kernel 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Installed-Size: 21 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Maintainer: Ubuntu Kernel Team 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: amd64 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Source: linux-meta 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Version: 5.15.0.101.98 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Recommends: thermald 2024-03-23T20:41:09.366 INFO:teuthology.orchestra.run.smithi117.stdout:Description: Generic Linux kernel image 2024-03-23T20:41:09.367 INFO:teuthology.orchestra.run.smithi117.stdout: This package will always depend on the latest generic kernel image 2024-03-23T20:41:09.367 INFO:teuthology.orchestra.run.smithi117.stdout: available. 2024-03-23T20:41:09.367 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-23T20:41:09.367 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-23T20:41:09.367 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-23T20:41:09.367 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-03-23T20:41:09.367 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-03-23T20:41:09.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get clean 2024-03-23T20:41:09.458 DEBUG:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2024-03-23T20:41:09.477 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2024-03-23T20:41:09.477 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 21 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 5.15.0.101.98 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2024-03-23T20:41:09.478 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2024-03-23T20:41:09.479 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2024-03-23T20:41:09.479 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-23T20:41:09.479 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-23T20:41:09.479 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-23T20:41:09.479 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-03-23T20:41:09.479 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-03-23T20:41:09.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2024-03-23T20:41:09.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2024-03-23T20:41:09.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2024-03-23T20:41:09.851 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-23T20:41:09.851 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-23T20:41:09.877 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-23T20:41:09.913 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-23T20:41:09.916 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-23T20:41:09.928 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-23T20:41:09.971 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-23T20:41:09.995 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-23T20:41:11.153 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-03-23T20:41:11.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-23T20:41:11.231 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-03-23T20:41:11.256 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-03-23T20:41:11.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-23T20:41:11.332 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-03-23T20:41:11.428 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2024-03-23T20:41:11.429 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2024-03-23T20:41:11.552 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2024-03-23T20:41:11.552 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2024-03-23T20:41:11.596 INFO:teuthology.orchestra.run.smithi204.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-23T20:41:11.596 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2024-03-23T20:41:11.596 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-23T20:41:11.596 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-23T20:41:11.597 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-23T20:41:11.597 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-23T20:41:11.597 INFO:teuthology.orchestra.run.smithi204.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-23T20:41:11.597 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-23T20:41:11.634 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-23T20:41:11.636 DEBUG:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2024-03-23T20:41:11.648 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2024-03-23T20:41:11.648 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2024-03-23T20:41:11.648 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2024-03-23T20:41:11.648 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2024-03-23T20:41:11.648 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 21 2024-03-23T20:41:11.648 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2024-03-23T20:41:11.648 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2024-03-23T20:41:11.649 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2024-03-23T20:41:11.649 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 5.15.0.101.98 2024-03-23T20:41:11.649 INFO:teuthology.orchestra.run.smithi204.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-23T20:41:11.649 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-23T20:41:11.649 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2024-03-23T20:41:11.649 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2024-03-23T20:41:11.649 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2024-03-23T20:41:11.649 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2024-03-23T20:41:11.649 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-23T20:41:11.694 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.uESKOO25pX 2024-03-23T20:41:11.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub/grub.cfg /tmp/tmp.uESKOO25pX 2024-03-23T20:41:11.736 INFO:teuthology.orchestra.run.smithi117.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-23T20:41:11.736 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2024-03-23T20:41:11.736 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-23T20:41:11.736 INFO:teuthology.orchestra.run.smithi117.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-23T20:41:11.737 INFO:teuthology.orchestra.run.smithi117.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-23T20:41:11.737 INFO:teuthology.orchestra.run.smithi117.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-23T20:41:11.737 INFO:teuthology.orchestra.run.smithi117.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-23T20:41:11.737 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-23T20:41:11.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.uESKOO25pX 2024-03-23T20:41:11.780 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-23T20:41:11.782 DEBUG:teuthology.orchestra.run.smithi117:> dpkg -s linux-image-generic 2024-03-23T20:41:11.845 INFO:teuthology.orchestra.run.smithi117.stdout:Package: linux-image-generic 2024-03-23T20:41:11.845 INFO:teuthology.orchestra.run.smithi117.stdout:Status: install ok installed 2024-03-23T20:41:11.845 INFO:teuthology.orchestra.run.smithi117.stdout:Priority: optional 2024-03-23T20:41:11.845 INFO:teuthology.orchestra.run.smithi117.stdout:Section: kernel 2024-03-23T20:41:11.845 INFO:teuthology.orchestra.run.smithi117.stdout:Installed-Size: 21 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout:Maintainer: Ubuntu Kernel Team 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: amd64 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout:Source: linux-meta 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout:Version: 5.15.0.101.98 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout:Recommends: thermald 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout:Description: Generic Linux kernel image 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout: This package will always depend on the latest generic kernel image 2024-03-23T20:41:11.846 INFO:teuthology.orchestra.run.smithi117.stdout: available. 2024-03-23T20:41:11.846 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-23T20:41:11.858 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.uESKOO25pX is 10KB 2024-03-23T20:41:11.870 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.uESKOO25pX 2024-03-23T20:41:11.875 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-23T20:41:11.890 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.tMKynrwAan 2024-03-23T20:41:11.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tMKynrwAan 2024-03-23T20:41:11.930 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:41:11.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-23T20:41:11.930 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-23T20:41:11.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.tMKynrwAan 2024-03-23T20:41:11.992 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-23T20:41:11.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo update-grub 2024-03-23T20:41:12.063 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.tMKynrwAan is 10KB 2024-03-23T20:41:12.076 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.tMKynrwAan 2024-03-23T20:41:12.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-23T20:41:12.133 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:41:12.133 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-23T20:41:12.133 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-23T20:41:12.197 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-23T20:41:12.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo update-grub 2024-03-23T20:41:13.207 INFO:teuthology.orchestra.run.smithi204.stderr:Sourcing file `/etc/default/grub' 2024-03-23T20:41:13.208 INFO:teuthology.orchestra.run.smithi204.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-23T20:41:13.209 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-03-23T20:41:13.381 INFO:teuthology.orchestra.run.smithi117.stderr:Sourcing file `/etc/default/grub' 2024-03-23T20:41:13.382 INFO:teuthology.orchestra.run.smithi117.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-23T20:41:13.385 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-03-23T20:41:13.392 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-23T20:41:13.398 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-23T20:41:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-23T20:41:13.583 INFO:teuthology.orchestra.run.smithi117.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-23T20:41:13.615 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-23T20:41:13.621 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-23T20:41:13.663 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-23T20:41:13.668 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-23T20:41:13.779 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-23T20:41:13.779 INFO:teuthology.orchestra.run.smithi204.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-23T20:41:13.779 INFO:teuthology.orchestra.run.smithi204.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-23T20:41:13.797 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-03-23T20:41:13.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-03-23T20:41:13.826 INFO:teuthology.orchestra.run.smithi117.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-23T20:41:13.832 INFO:teuthology.orchestra.run.smithi117.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-23T20:41:13.872 INFO:teuthology.orchestra.run.smithi117.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-23T20:41:13.877 INFO:teuthology.orchestra.run.smithi117.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-23T20:41:14.016 INFO:teuthology.orchestra.run.smithi117.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-23T20:41:14.016 INFO:teuthology.orchestra.run.smithi117.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-23T20:41:14.016 INFO:teuthology.orchestra.run.smithi117.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-23T20:41:14.027 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-03-23T20:41:14.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-03-23T20:41:43.828 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-03-23T20:41:43.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-23T20:41:43.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:41:44.043 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-03-23T20:41:44.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-23T20:41:44.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:42:02.399 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-23T20:42:02.399 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-23T20:42:11.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-23T20:42:11.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:42:11.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-23T20:42:11.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:42:14.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-23T20:42:14.463 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-23T20:42:26.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-23T20:42:26.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:42:26.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-23T20:42:26.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:42:44.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-23T20:42:59.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-23T20:42:59.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:43:00.137 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-23T20:43:00.853 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-23T20:43:00.853 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-101-generic"... 2024-03-23T20:43:00.854 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-23T20:43:00.859 INFO:teuthology.orchestra.run.smithi117.stdout:5.15.0-101-generic 2024-03-23T20:43:00.859 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-23T20:43:00.859 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T20:43:00.859 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-23T20:43:01.861 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-03-23T20:43:01.861 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T20:43:01.946 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-03-23T20:43:01.968 DEBUG:teuthology.parallel:result is None 2024-03-23T20:43:26.467 DEBUG:teuthology.orchestra.remote:timed out 2024-03-23T20:43:41.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-23T20:43:41.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:43:41.692 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-23T20:43:42.264 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-23T20:43:42.264 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-101-generic"... 2024-03-23T20:43:42.264 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-23T20:43:42.310 INFO:teuthology.orchestra.run.smithi204.stdout:5.15.0-101-generic 2024-03-23T20:43:42.310 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-23T20:43:42.310 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T20:43:42.311 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-23T20:43:43.311 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-03-23T20:43:43.312 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T20:43:43.653 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-03-23T20:43:43.675 DEBUG:teuthology.parallel:result is None 2024-03-23T20:43:43.676 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-23T20:43:43.683 INFO:teuthology.task.internal:Creating test directory... 2024-03-23T20:43:43.684 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T20:43:43.686 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T20:43:43.692 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-23T20:43:43.698 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-23T20:43:43.706 INFO:teuthology.task.internal:Creating archive directory... 2024-03-23T20:43:43.706 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T20:43:43.734 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T20:43:43.766 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-23T20:43:43.773 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-23T20:43:43.773 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T20:43:43.790 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T20:43:43.810 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T20:43:43.813 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T20:43:43.818 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T20:43:43.821 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T20:43:43.823 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-23T20:43:43.830 INFO:teuthology.task.internal:Configuring sudo... 2024-03-23T20:43:43.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T20:43:43.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T20:43:43.882 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-23T20:43:43.892 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-23T20:43:43.892 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T20:43:43.926 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T20:43:43.932 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T20:43:43.977 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T20:43:44.025 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:43:44.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T20:43:44.087 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T20:43:44.092 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T20:43:44.139 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:43:44.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T20:43:44.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-03-23T20:43:44.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-03-23T20:43:44.274 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-23T20:43:44.281 INFO:teuthology.task.internal:Starting timer... 2024-03-23T20:43:44.282 INFO:teuthology.run_tasks:Running task pcp... 2024-03-23T20:43:44.291 INFO:teuthology.run_tasks:Running task selinux... 2024-03-23T20:43:44.300 DEBUG:teuthology.task.selinux:Excluding smithi117: OS 'ubuntu' does not support SELinux 2024-03-23T20:43:44.300 DEBUG:teuthology.task.selinux:Excluding smithi204: OS 'ubuntu' does not support SELinux 2024-03-23T20:43:44.300 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-23T20:43:44.300 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-23T20:43:44.301 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-23T20:43:44.301 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-23T20:43:44.309 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-23T20:43:44.326 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-23T20:43:44.327 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi117.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-23T20:50:51.915 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-03-23T20:50:51.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-23T20:50:51.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:50:52.003 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-23T20:50:52.082 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-23T20:50:52.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-23T20:50:52.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T20:50:52.166 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-23T20:50:52.242 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-23T20:50:52.242 INFO:teuthology.run_tasks:Running task clock... 2024-03-23T20:50:52.252 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-23T20:50:52.252 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T20:50:52.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T20:50:52.255 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T20:50:52.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T20:50:52.281 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-23T20:50:52.282 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Command line: ntpd -gq 2024-03-23T20:50:52.282 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: ---------------------------------------------------- 2024-03-23T20:50:52.282 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: ntp-4 is maintained by Network Time Foundation, 2024-03-23T20:50:52.282 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-23T20:50:52.282 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: corporation. Support and training for ntp-4 are 2024-03-23T20:50:52.282 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: available at https://www.nwtime.org/support 2024-03-23T20:50:52.282 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: ---------------------------------------------------- 2024-03-23T20:50:52.283 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: proto: precision = 0.043 usec (-24) 2024-03-23T20:50:52.283 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: basedate set to 2022-02-04 2024-03-23T20:50:52.283 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: gps base set to 2022-02-06 (week 2196) 2024-03-23T20:50:52.283 INFO:teuthology.orchestra.run.smithi117.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-23T20:50:52.284 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-23T20:50:52.284 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: restrict ::: KOD does nothing without LIMITED. 2024-03-23T20:50:52.284 INFO:teuthology.orchestra.run.smithi117.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-23T20:50:52.285 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Listen and drop on 0 v6wildcard [::]:123 2024-03-23T20:50:52.285 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-23T20:50:52.285 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Listen normally on 2 lo 127.0.0.1:123 2024-03-23T20:50:52.285 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Listen normally on 3 enp3s0f1 172.21.15.117:123 2024-03-23T20:50:52.285 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Listen normally on 4 lo [::1]:123 2024-03-23T20:50:52.285 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4463%5]:123 2024-03-23T20:50:52.285 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:52 ntpd[17208]: Listening on routing socket on fd #22 for interface updates 2024-03-23T20:50:52.311 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-23T20:50:52.312 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Command line: ntpd -gq 2024-03-23T20:50:52.312 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: ---------------------------------------------------- 2024-03-23T20:50:52.312 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: ntp-4 is maintained by Network Time Foundation, 2024-03-23T20:50:52.312 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-23T20:50:52.312 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: corporation. Support and training for ntp-4 are 2024-03-23T20:50:52.312 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: available at https://www.nwtime.org/support 2024-03-23T20:50:52.312 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: ---------------------------------------------------- 2024-03-23T20:50:52.313 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: proto: precision = 0.045 usec (-24) 2024-03-23T20:50:52.313 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: basedate set to 2022-02-04 2024-03-23T20:50:52.313 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: gps base set to 2022-02-06 (week 2196) 2024-03-23T20:50:52.313 INFO:teuthology.orchestra.run.smithi204.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-23T20:50:52.313 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-23T20:50:52.314 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: restrict ::: KOD does nothing without LIMITED. 2024-03-23T20:50:52.314 INFO:teuthology.orchestra.run.smithi204.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-23T20:50:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Listen and drop on 0 v6wildcard [::]:123 2024-03-23T20:50:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-23T20:50:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Listen normally on 2 lo 127.0.0.1:123 2024-03-23T20:50:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Listen normally on 3 enp3s0f1 172.21.15.204:123 2024-03-23T20:50:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Listen normally on 4 lo [::1]:123 2024-03-23T20:50:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77ab%5]:123 2024-03-23T20:50:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:52 ntpd[17172]: Listening on routing socket on fd #22 for interface updates 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:53.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:53 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:53.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:53.314 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:53 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:54.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:54 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:54.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:54 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:54.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:54 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:54.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:54 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:54.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:54 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:54.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:54 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:55.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:55.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:55.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:55.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:55.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:55 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:55.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:55.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:55 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:56.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:56 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:56.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:56 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:56.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:56 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:56.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:56 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:56.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:56 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:56.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:56 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:57.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:57 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:57.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:57 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:58.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:58 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:58.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:58 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:58.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:58 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:58.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:58 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:58.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:58 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:58.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:58 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:59.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:59.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:59.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:59.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:59.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:59.283 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stdout:23 Mar 20:50:59 ntpd[17208]: ntpd: time slew -0.000570 s 2024-03-23T20:50:59.284 INFO:teuthology.orchestra.run.smithi117.stdout:ntpd: time slew -0.000570s 2024-03-23T20:50:59.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-23T20:50:59.285 INFO:teuthology.orchestra.run.smithi117.stderr:23 Mar 20:50:59 ntpd[17208]: can't open /var/log/ntpstats/loopstats.20240323: Permission denied 2024-03-23T20:50:59.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:59.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:59.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:59.312 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:59.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-23T20:50:59.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: can't open /var/log/ntpstats/rawstats.20240323: Permission denied 2024-03-23T20:50:59.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-23T20:50:59.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: can't open /var/log/ntpstats/peerstats.20240323: Permission denied 2024-03-23T20:50:59.313 INFO:teuthology.orchestra.run.smithi204.stdout:23 Mar 20:50:59 ntpd[17172]: ntpd: time slew -0.000536 s 2024-03-23T20:50:59.313 INFO:teuthology.orchestra.run.smithi204.stdout:ntpd: time slew -0.000536s 2024-03-23T20:50:59.313 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-23T20:50:59.314 INFO:teuthology.orchestra.run.smithi204.stderr:23 Mar 20:50:59 ntpd[17172]: can't open /var/log/ntpstats/loopstats.20240323: Permission denied 2024-03-23T20:50:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: remote refid st t when poll reach delay offset jitter 2024-03-23T20:50:59.331 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================== 2024-03-23T20:50:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-23T20:50:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-23T20:50:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-23T20:50:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-23T20:50:59.422 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2024-03-23T20:50:59.422 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2024-03-23T20:50:59.422 INFO:teuthology.orchestra.run.smithi204.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-23T20:50:59.422 INFO:teuthology.orchestra.run.smithi204.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-23T20:50:59.422 INFO:teuthology.orchestra.run.smithi204.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-23T20:50:59.422 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-23T20:50:59.423 INFO:teuthology.run_tasks:Running task install... 2024-03-23T20:50:59.436 DEBUG:teuthology.task.install:project ceph 2024-03-23T20:50:59.436 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}} 2024-03-23T20:50:59.436 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'} 2024-03-23T20:50:59.436 INFO:teuthology.task.install:Using flavor: default 2024-03-23T20:50:59.446 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-23T20:50:59.446 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-03-23T20:50:59.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-key list | grep Ceph 2024-03-23T20:50:59.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-key list | grep Ceph 2024-03-23T20:50:59.509 INFO:teuthology.orchestra.run.smithi117.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-23T20:50:59.524 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-23T20:50:59.537 INFO:teuthology.orchestra.run.smithi117.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-23T20:50:59.538 INFO:teuthology.orchestra.run.smithi117.stdout:uid [ unknown] Ceph.com (release key) 2024-03-23T20:50:59.538 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-03-23T20:50:59.538 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T20:50:59.551 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-23T20:50:59.552 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph.com (release key) 2024-03-23T20:50:59.552 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-03-23T20:50:59.552 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T20:50:59.755 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default/ 2024-03-23T20:50:59.755 INFO:teuthology.task.install.deb:Package version is 18.2.2-954-g82316b04-1jammy 2024-03-23T20:50:59.768 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default/ 2024-03-23T20:50:59.768 INFO:teuthology.task.install.deb:Package version is 18.2.2-954-g82316b04-1jammy 2024-03-23T20:50:59.874 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:50:59.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-23T20:50:59.892 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:50:59.892 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-23T20:50:59.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2024-03-23T20:50:59.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2024-03-23T20:51:00.067 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-23T20:51:00.068 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-23T20:51:00.079 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-23T20:51:00.079 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-23T20:51:00.079 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-23T20:51:00.091 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-23T20:51:00.098 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-23T20:51:00.112 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-23T20:51:00.185 INFO:teuthology.orchestra.run.smithi117.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy InRelease 2024-03-23T20:51:00.188 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy InRelease 2024-03-23T20:51:00.221 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-23T20:51:00.224 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-23T20:51:00.258 INFO:teuthology.orchestra.run.smithi117.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-23T20:51:00.261 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-23T20:51:00.294 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2024-03-23T20:51:00.297 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-03-23T20:51:00.336 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-03-23T20:51:00.339 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2024-03-23T20:51:00.425 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 28.5 kB in 0s (83.1 kB/s) 2024-03-23T20:51:00.425 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 28.5 kB in 0s (81.4 kB/s) 2024-03-23T20:51:01.349 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-03-23T20:51:01.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-954-g82316b04-1jammy cephadm=18.2.2-954-g82316b04-1jammy ceph-mds=18.2.2-954-g82316b04-1jammy ceph-mgr=18.2.2-954-g82316b04-1jammy ceph-common=18.2.2-954-g82316b04-1jammy ceph-fuse=18.2.2-954-g82316b04-1jammy ceph-test=18.2.2-954-g82316b04-1jammy ceph-volume=18.2.2-954-g82316b04-1jammy radosgw=18.2.2-954-g82316b04-1jammy python3-rados=18.2.2-954-g82316b04-1jammy python3-rgw=18.2.2-954-g82316b04-1jammy python3-cephfs=18.2.2-954-g82316b04-1jammy python3-rbd=18.2.2-954-g82316b04-1jammy libcephfs2=18.2.2-954-g82316b04-1jammy libcephfs-dev=18.2.2-954-g82316b04-1jammy librados2=18.2.2-954-g82316b04-1jammy librbd1=18.2.2-954-g82316b04-1jammy rbd-fuse=18.2.2-954-g82316b04-1jammy rbd-mirror=18.2.2-954-g82316b04-1jammy 2024-03-23T20:51:01.400 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-03-23T20:51:01.415 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2024-03-23T20:51:01.415 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-954-g82316b04-1jammy cephadm=18.2.2-954-g82316b04-1jammy ceph-mds=18.2.2-954-g82316b04-1jammy ceph-mgr=18.2.2-954-g82316b04-1jammy ceph-common=18.2.2-954-g82316b04-1jammy ceph-fuse=18.2.2-954-g82316b04-1jammy ceph-test=18.2.2-954-g82316b04-1jammy ceph-volume=18.2.2-954-g82316b04-1jammy radosgw=18.2.2-954-g82316b04-1jammy python3-rados=18.2.2-954-g82316b04-1jammy python3-rgw=18.2.2-954-g82316b04-1jammy python3-cephfs=18.2.2-954-g82316b04-1jammy python3-rbd=18.2.2-954-g82316b04-1jammy libcephfs2=18.2.2-954-g82316b04-1jammy libcephfs-dev=18.2.2-954-g82316b04-1jammy librados2=18.2.2-954-g82316b04-1jammy librbd1=18.2.2-954-g82316b04-1jammy rbd-fuse=18.2.2-954-g82316b04-1jammy rbd-mirror=18.2.2-954-g82316b04-1jammy 2024-03-23T20:51:01.468 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-03-23T20:51:01.587 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2024-03-23T20:51:01.587 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2024-03-23T20:51:01.641 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2024-03-23T20:51:01.642 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2024-03-23T20:51:01.744 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2024-03-23T20:51:01.744 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-23T20:51:01.744 INFO:teuthology.orchestra.run.smithi117.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-23T20:51:01.744 INFO:teuthology.orchestra.run.smithi117.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-23T20:51:01.745 INFO:teuthology.orchestra.run.smithi117.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-23T20:51:01.745 INFO:teuthology.orchestra.run.smithi117.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-23T20:51:01.745 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-23T20:51:01.745 INFO:teuthology.orchestra.run.smithi117.stdout:The following additional packages will be installed: 2024-03-23T20:51:01.746 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-23T20:51:01.746 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-23T20:51:01.746 INFO:teuthology.orchestra.run.smithi117.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-03-23T20:51:01.746 INFO:teuthology.orchestra.run.smithi117.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-23T20:51:01.746 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-23T20:51:01.747 INFO:teuthology.orchestra.run.smithi117.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-03-23T20:51:01.748 INFO:teuthology.orchestra.run.smithi117.stdout: python3-watchdog 2024-03-23T20:51:01.749 INFO:teuthology.orchestra.run.smithi117.stdout:Recommended packages: 2024-03-23T20:51:01.749 INFO:teuthology.orchestra.run.smithi117.stdout: btrfs-tools 2024-03-23T20:51:01.799 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2024-03-23T20:51:01.799 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-23T20:51:01.799 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-23T20:51:01.799 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-23T20:51:01.799 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-23T20:51:01.800 INFO:teuthology.orchestra.run.smithi204.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-23T20:51:01.800 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-23T20:51:01.800 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2024-03-23T20:51:01.800 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-23T20:51:01.801 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-23T20:51:01.801 INFO:teuthology.orchestra.run.smithi204.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-03-23T20:51:01.801 INFO:teuthology.orchestra.run.smithi204.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-23T20:51:01.801 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-23T20:51:01.802 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-03-23T20:51:01.803 INFO:teuthology.orchestra.run.smithi204.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-03-23T20:51:01.804 INFO:teuthology.orchestra.run.smithi204.stdout: python3-watchdog 2024-03-23T20:51:01.804 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2024-03-23T20:51:01.804 INFO:teuthology.orchestra.run.smithi204.stdout: btrfs-tools 2024-03-23T20:51:01.846 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2024-03-23T20:51:01.846 INFO:teuthology.orchestra.run.smithi117.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-23T20:51:01.847 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-23T20:51:01.847 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-23T20:51:01.847 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-03-23T20:51:01.847 INFO:teuthology.orchestra.run.smithi117.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-23T20:51:01.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-23T20:51:01.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-23T20:51:01.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-23T20:51:01.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-23T20:51:01.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-23T20:51:01.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-23T20:51:01.849 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror socat xmlstarlet 2024-03-23T20:51:01.901 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2024-03-23T20:51:01.901 INFO:teuthology.orchestra.run.smithi204.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-23T20:51:01.901 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-23T20:51:01.901 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-23T20:51:01.902 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-03-23T20:51:01.902 INFO:teuthology.orchestra.run.smithi204.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-23T20:51:01.903 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-23T20:51:01.904 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-23T20:51:01.904 INFO:teuthology.orchestra.run.smithi204.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-23T20:51:01.904 INFO:teuthology.orchestra.run.smithi204.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-23T20:51:01.904 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-23T20:51:01.904 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-23T20:51:01.904 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror socat xmlstarlet 2024-03-23T20:51:01.912 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 110 newly installed, 0 to remove and 285 not upgraded. 2024-03-23T20:51:01.912 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 165 MB of archives. 2024-03-23T20:51:01.913 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 674 MB of additional disk space will be used. 2024-03-23T20:51:01.913 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-03-23T20:51:01.970 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 110 newly installed, 0 to remove and 285 not upgraded. 2024-03-23T20:51:01.970 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 165 MB of archives. 2024-03-23T20:51:01.970 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 674 MB of additional disk space will be used. 2024-03-23T20:51:01.970 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-03-23T20:51:01.981 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-23T20:51:02.025 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-23T20:51:02.028 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-23T20:51:02.036 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-23T20:51:02.047 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-23T20:51:02.065 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-954-g82316b04-1jammy [3,418 kB] 2024-03-23T20:51:02.078 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-23T20:51:02.081 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-23T20:51:02.098 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-23T20:51:02.104 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-23T20:51:02.109 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-23T20:51:02.121 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-23T20:51:02.122 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-954-g82316b04-1jammy [3,418 kB] 2024-03-23T20:51:02.125 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-23T20:51:02.125 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-23T20:51:02.126 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-23T20:51:02.135 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-23T20:51:02.135 INFO:teuthology.orchestra.run.smithi117.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-23T20:51:02.135 INFO:teuthology.orchestra.run.smithi117.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-23T20:51:02.137 INFO:teuthology.orchestra.run.smithi117.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-23T20:51:02.150 INFO:teuthology.orchestra.run.smithi117.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-23T20:51:02.151 INFO:teuthology.orchestra.run.smithi117.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-23T20:51:02.151 INFO:teuthology.orchestra.run.smithi117.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-23T20:51:02.151 INFO:teuthology.orchestra.run.smithi117.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-23T20:51:02.157 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-23T20:51:02.161 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-23T20:51:02.171 INFO:teuthology.orchestra.run.smithi117.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-23T20:51:02.171 INFO:teuthology.orchestra.run.smithi117.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-23T20:51:02.173 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-23T20:51:02.174 INFO:teuthology.orchestra.run.smithi117.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-23T20:51:02.174 INFO:teuthology.orchestra.run.smithi117.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-23T20:51:02.175 INFO:teuthology.orchestra.run.smithi117.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-23T20:51:02.175 INFO:teuthology.orchestra.run.smithi117.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-23T20:51:02.178 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-23T20:51:02.178 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-23T20:51:02.179 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-23T20:51:02.187 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-23T20:51:02.188 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-23T20:51:02.188 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-23T20:51:02.189 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-23T20:51:02.191 INFO:teuthology.orchestra.run.smithi117.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-23T20:51:02.192 INFO:teuthology.orchestra.run.smithi117.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-23T20:51:02.192 INFO:teuthology.orchestra.run.smithi117.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-23T20:51:02.193 INFO:teuthology.orchestra.run.smithi117.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-23T20:51:02.200 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-23T20:51:02.200 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-23T20:51:02.201 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-23T20:51:02.213 INFO:teuthology.orchestra.run.smithi117.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-23T20:51:02.213 INFO:teuthology.orchestra.run.smithi117.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-23T20:51:02.218 INFO:teuthology.orchestra.run.smithi117.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-23T20:51:02.219 INFO:teuthology.orchestra.run.smithi117.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-23T20:51:02.219 INFO:teuthology.orchestra.run.smithi117.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-23T20:51:02.220 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-23T20:51:02.220 INFO:teuthology.orchestra.run.smithi117.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-23T20:51:02.220 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-23T20:51:02.221 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-23T20:51:02.223 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-23T20:51:02.223 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-23T20:51:02.224 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-23T20:51:02.224 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-23T20:51:02.233 INFO:teuthology.orchestra.run.smithi117.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-23T20:51:02.235 INFO:teuthology.orchestra.run.smithi117.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-23T20:51:02.236 INFO:teuthology.orchestra.run.smithi117.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-23T20:51:02.237 INFO:teuthology.orchestra.run.smithi117.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-23T20:51:02.239 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-23T20:51:02.240 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-23T20:51:02.240 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-23T20:51:02.257 INFO:teuthology.orchestra.run.smithi117.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-23T20:51:02.259 INFO:teuthology.orchestra.run.smithi117.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-23T20:51:02.260 INFO:teuthology.orchestra.run.smithi117.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-23T20:51:02.262 INFO:teuthology.orchestra.run.smithi117.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-23T20:51:02.272 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-23T20:51:02.274 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-23T20:51:02.274 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-23T20:51:02.285 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-23T20:51:02.286 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-23T20:51:02.287 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-23T20:51:02.288 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-23T20:51:02.319 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-23T20:51:02.323 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-23T20:51:02.325 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-23T20:51:02.338 INFO:teuthology.orchestra.run.smithi117.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-23T20:51:02.339 INFO:teuthology.orchestra.run.smithi117.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-23T20:51:02.341 INFO:teuthology.orchestra.run.smithi117.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-23T20:51:02.342 INFO:teuthology.orchestra.run.smithi117.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-23T20:51:02.342 INFO:teuthology.orchestra.run.smithi117.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-23T20:51:02.343 INFO:teuthology.orchestra.run.smithi117.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-23T20:51:02.344 INFO:teuthology.orchestra.run.smithi117.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-23T20:51:02.345 INFO:teuthology.orchestra.run.smithi117.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-23T20:51:02.348 INFO:teuthology.orchestra.run.smithi117.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-23T20:51:02.357 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-23T20:51:02.358 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-23T20:51:02.363 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-23T20:51:02.364 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-23T20:51:02.369 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-23T20:51:02.394 INFO:teuthology.orchestra.run.smithi117.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-23T20:51:02.396 INFO:teuthology.orchestra.run.smithi117.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-23T20:51:02.452 INFO:teuthology.orchestra.run.smithi117.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-954-g82316b04-1jammy [3,105 kB] 2024-03-23T20:51:02.478 INFO:teuthology.orchestra.run.smithi117.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-23T20:51:02.481 INFO:teuthology.orchestra.run.smithi117.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-23T20:51:02.481 INFO:teuthology.orchestra.run.smithi117.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-23T20:51:02.502 INFO:teuthology.orchestra.run.smithi204.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-23T20:51:02.503 INFO:teuthology.orchestra.run.smithi204.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-23T20:51:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-23T20:51:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-23T20:51:02.505 INFO:teuthology.orchestra.run.smithi204.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-23T20:51:02.506 INFO:teuthology.orchestra.run.smithi117.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-23T20:51:02.506 INFO:teuthology.orchestra.run.smithi204.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-23T20:51:02.506 INFO:teuthology.orchestra.run.smithi117.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-23T20:51:02.507 INFO:teuthology.orchestra.run.smithi204.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-23T20:51:02.507 INFO:teuthology.orchestra.run.smithi117.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-23T20:51:02.507 INFO:teuthology.orchestra.run.smithi117.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-23T20:51:02.508 INFO:teuthology.orchestra.run.smithi204.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-23T20:51:02.508 INFO:teuthology.orchestra.run.smithi117.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-23T20:51:02.508 INFO:teuthology.orchestra.run.smithi204.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-23T20:51:02.509 INFO:teuthology.orchestra.run.smithi204.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-23T20:51:02.512 INFO:teuthology.orchestra.run.smithi204.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-23T20:51:02.523 INFO:teuthology.orchestra.run.smithi117.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-23T20:51:02.527 INFO:teuthology.orchestra.run.smithi204.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-23T20:51:02.528 INFO:teuthology.orchestra.run.smithi117.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-23T20:51:02.533 INFO:teuthology.orchestra.run.smithi204.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-23T20:51:02.560 INFO:teuthology.orchestra.run.smithi204.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-23T20:51:02.563 INFO:teuthology.orchestra.run.smithi204.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-23T20:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-23T20:51:02.563 INFO:teuthology.orchestra.run.smithi204.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-23T20:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-23T20:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-23T20:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-23T20:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-23T20:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-23T20:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-954-g82316b04-1jammy [674 kB] 2024-03-23T20:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-23T20:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-23T20:51:02.596 INFO:teuthology.orchestra.run.smithi204.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-23T20:51:02.596 INFO:teuthology.orchestra.run.smithi204.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-23T20:51:02.596 INFO:teuthology.orchestra.run.smithi204.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-23T20:51:02.597 INFO:teuthology.orchestra.run.smithi204.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-23T20:51:02.597 INFO:teuthology.orchestra.run.smithi204.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-23T20:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-23T20:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-954-g82316b04-1jammy [356 kB] 2024-03-23T20:51:02.612 INFO:teuthology.orchestra.run.smithi204.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-23T20:51:02.613 INFO:teuthology.orchestra.run.smithi204.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-23T20:51:02.613 INFO:teuthology.orchestra.run.smithi204.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-23T20:51:02.614 INFO:teuthology.orchestra.run.smithi117.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-23T20:51:02.618 INFO:teuthology.orchestra.run.smithi204.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-23T20:51:02.618 INFO:teuthology.orchestra.run.smithi117.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-954-g82316b04-1jammy [32.8 kB] 2024-03-23T20:51:02.619 INFO:teuthology.orchestra.run.smithi117.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-954-g82316b04-1jammy [180 kB] 2024-03-23T20:51:02.623 INFO:teuthology.orchestra.run.smithi117.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-954-g82316b04-1jammy [62.5 kB] 2024-03-23T20:51:02.625 INFO:teuthology.orchestra.run.smithi117.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-954-g82316b04-1jammy [330 kB] 2024-03-23T20:51:02.627 INFO:teuthology.orchestra.run.smithi117.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-23T20:51:02.628 INFO:teuthology.orchestra.run.smithi117.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-23T20:51:02.635 INFO:teuthology.orchestra.run.smithi117.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-954-g82316b04-1jammy [5,889 kB] 2024-03-23T20:51:02.655 INFO:teuthology.orchestra.run.smithi204.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-23T20:51:02.657 INFO:teuthology.orchestra.run.smithi204.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-23T20:51:02.658 INFO:teuthology.orchestra.run.smithi204.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-23T20:51:02.664 INFO:teuthology.orchestra.run.smithi204.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-23T20:51:02.668 INFO:teuthology.orchestra.run.smithi204.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-23T20:51:02.675 INFO:teuthology.orchestra.run.smithi204.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-23T20:51:02.687 INFO:teuthology.orchestra.run.smithi204.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-23T20:51:02.688 INFO:teuthology.orchestra.run.smithi204.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-23T20:51:02.691 INFO:teuthology.orchestra.run.smithi204.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-23T20:51:02.694 INFO:teuthology.orchestra.run.smithi204.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-954-g82316b04-1jammy [3,105 kB] 2024-03-23T20:51:02.702 INFO:teuthology.orchestra.run.smithi204.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-23T20:51:02.707 INFO:teuthology.orchestra.run.smithi204.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-23T20:51:02.708 INFO:teuthology.orchestra.run.smithi204.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-23T20:51:02.881 INFO:teuthology.orchestra.run.smithi117.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-954-g82316b04-1jammy [112 kB] 2024-03-23T20:51:02.884 INFO:teuthology.orchestra.run.smithi117.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-954-g82316b04-1jammy [441 kB] 2024-03-23T20:51:02.897 INFO:teuthology.orchestra.run.smithi117.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-954-g82316b04-1jammy [22.0 MB] 2024-03-23T20:51:03.022 INFO:teuthology.orchestra.run.smithi204.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-954-g82316b04-1jammy [674 kB] 2024-03-23T20:51:03.111 INFO:teuthology.orchestra.run.smithi204.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-954-g82316b04-1jammy [356 kB] 2024-03-23T20:51:03.191 INFO:teuthology.orchestra.run.smithi204.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-954-g82316b04-1jammy [32.8 kB] 2024-03-23T20:51:03.193 INFO:teuthology.orchestra.run.smithi204.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-954-g82316b04-1jammy [180 kB] 2024-03-23T20:51:03.199 INFO:teuthology.orchestra.run.smithi204.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-954-g82316b04-1jammy [62.5 kB] 2024-03-23T20:51:03.201 INFO:teuthology.orchestra.run.smithi204.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-954-g82316b04-1jammy [330 kB] 2024-03-23T20:51:03.304 INFO:teuthology.orchestra.run.smithi204.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-954-g82316b04-1jammy [5,889 kB] 2024-03-23T20:51:03.803 INFO:teuthology.orchestra.run.smithi204.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-954-g82316b04-1jammy [112 kB] 2024-03-23T20:51:03.806 INFO:teuthology.orchestra.run.smithi204.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-954-g82316b04-1jammy [441 kB] 2024-03-23T20:51:03.862 INFO:teuthology.orchestra.run.smithi204.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-954-g82316b04-1jammy [22.0 MB] 2024-03-23T20:51:04.067 INFO:teuthology.orchestra.run.smithi117.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-954-g82316b04-1jammy [5,002 kB] 2024-03-23T20:51:04.232 INFO:teuthology.orchestra.run.smithi117.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-954-g82316b04-1jammy [241 kB] 2024-03-23T20:51:04.239 INFO:teuthology.orchestra.run.smithi117.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-954-g82316b04-1jammy [130 kB] 2024-03-23T20:51:04.243 INFO:teuthology.orchestra.run.smithi117.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-954-g82316b04-1jammy [1,485 kB] 2024-03-23T20:51:04.314 INFO:teuthology.orchestra.run.smithi117.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-954-g82316b04-1jammy [6,078 kB] 2024-03-23T20:51:04.606 INFO:teuthology.orchestra.run.smithi117.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-954-g82316b04-1jammy [22.3 MB] 2024-03-23T20:51:05.744 INFO:teuthology.orchestra.run.smithi117.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-954-g82316b04-1jammy [14.1 kB] 2024-03-23T20:51:05.745 INFO:teuthology.orchestra.run.smithi117.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-954-g82316b04-1jammy [826 kB] 2024-03-23T20:51:05.760 INFO:teuthology.orchestra.run.smithi117.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-954-g82316b04-1jammy [2,173 kB] 2024-03-23T20:51:05.813 INFO:teuthology.orchestra.run.smithi117.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-954-g82316b04-1jammy [238 kB] 2024-03-23T20:51:05.821 INFO:teuthology.orchestra.run.smithi117.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-954-g82316b04-1jammy [144 kB] 2024-03-23T20:51:05.848 INFO:teuthology.orchestra.run.smithi117.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-954-g82316b04-1jammy [1,882 kB] 2024-03-23T20:51:05.934 INFO:teuthology.orchestra.run.smithi117.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-954-g82316b04-1jammy [8,625 kB] 2024-03-23T20:51:06.168 INFO:teuthology.orchestra.run.smithi204.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-954-g82316b04-1jammy [5,002 kB] 2024-03-23T20:51:06.339 INFO:teuthology.orchestra.run.smithi117.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-954-g82316b04-1jammy [14.3 kB] 2024-03-23T20:51:06.340 INFO:teuthology.orchestra.run.smithi117.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-954-g82316b04-1jammy [42.7 MB] 2024-03-23T20:51:06.662 INFO:teuthology.orchestra.run.smithi204.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-954-g82316b04-1jammy [241 kB] 2024-03-23T20:51:06.668 INFO:teuthology.orchestra.run.smithi204.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-954-g82316b04-1jammy [130 kB] 2024-03-23T20:51:06.673 INFO:teuthology.orchestra.run.smithi204.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-954-g82316b04-1jammy [1,485 kB] 2024-03-23T20:51:06.843 INFO:teuthology.orchestra.run.smithi204.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-954-g82316b04-1jammy [6,078 kB] 2024-03-23T20:51:07.575 INFO:teuthology.orchestra.run.smithi204.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-954-g82316b04-1jammy [22.3 MB] 2024-03-23T20:51:08.499 INFO:teuthology.orchestra.run.smithi117.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-954-g82316b04-1jammy [121 kB] 2024-03-23T20:51:08.502 INFO:teuthology.orchestra.run.smithi117.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-954-g82316b04-1jammy [38.3 kB] 2024-03-23T20:51:08.502 INFO:teuthology.orchestra.run.smithi117.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-954-g82316b04-1jammy [11.6 MB] 2024-03-23T20:51:08.930 INFO:teuthology.orchestra.run.smithi117.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-954-g82316b04-1jammy [93.1 kB] 2024-03-23T20:51:08.932 INFO:teuthology.orchestra.run.smithi117.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 18.2.2-954-g82316b04-1jammy [3,031 kB] 2024-03-23T20:51:09.493 INFO:teuthology.orchestra.run.smithi204.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-954-g82316b04-1jammy [14.1 kB] 2024-03-23T20:51:09.494 INFO:teuthology.orchestra.run.smithi204.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-954-g82316b04-1jammy [826 kB] 2024-03-23T20:51:09.509 INFO:teuthology.orchestra.run.smithi204.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-954-g82316b04-1jammy [2,173 kB] 2024-03-23T20:51:09.561 INFO:teuthology.orchestra.run.smithi204.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-954-g82316b04-1jammy [238 kB] 2024-03-23T20:51:09.566 INFO:teuthology.orchestra.run.smithi204.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-954-g82316b04-1jammy [144 kB] 2024-03-23T20:51:09.588 INFO:teuthology.orchestra.run.smithi204.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-954-g82316b04-1jammy [1,882 kB] 2024-03-23T20:51:09.631 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 165 MB in 7s (22.8 MB/s) 2024-03-23T20:51:09.665 INFO:teuthology.orchestra.run.smithi204.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-954-g82316b04-1jammy [8,625 kB] 2024-03-23T20:51:09.769 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libfmt8:amd64. 2024-03-23T20:51:09.866 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-23T20:51:09.871 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-03-23T20:51:09.959 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-03-23T20:51:10.033 INFO:teuthology.orchestra.run.smithi204.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-954-g82316b04-1jammy [14.3 kB] 2024-03-23T20:51:10.033 INFO:teuthology.orchestra.run.smithi204.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-954-g82316b04-1jammy [42.7 MB] 2024-03-23T20:51:10.295 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-23T20:51:10.322 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-23T20:51:10.362 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-23T20:51:10.674 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-23T20:51:10.700 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-23T20:51:10.740 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-23T20:51:11.027 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-23T20:51:11.053 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-23T20:51:11.093 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-23T20:51:11.530 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-23T20:51:11.556 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-23T20:51:11.605 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:51:11.991 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-23T20:51:12.017 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-23T20:51:12.057 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:51:12.067 INFO:teuthology.orchestra.run.smithi204.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-954-g82316b04-1jammy [121 kB] 2024-03-23T20:51:12.069 INFO:teuthology.orchestra.run.smithi204.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-954-g82316b04-1jammy [38.3 kB] 2024-03-23T20:51:12.070 INFO:teuthology.orchestra.run.smithi204.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-954-g82316b04-1jammy [11.6 MB] 2024-03-23T20:51:12.394 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-23T20:51:12.420 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-23T20:51:12.460 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:51:12.490 INFO:teuthology.orchestra.run.smithi204.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-954-g82316b04-1jammy [93.1 kB] 2024-03-23T20:51:12.492 INFO:teuthology.orchestra.run.smithi204.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 18.2.2-954-g82316b04-1jammy [3,031 kB] 2024-03-23T20:51:12.780 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-23T20:51:12.807 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-23T20:51:12.847 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-23T20:51:13.158 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librados2. 2024-03-23T20:51:13.169 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 165 MB in 11s (15.4 MB/s) 2024-03-23T20:51:13.184 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../008-librados2_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:13.241 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librados2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:13.304 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libfmt8:amd64. 2024-03-23T20:51:13.403 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-23T20:51:13.409 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-03-23T20:51:13.494 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-03-23T20:51:13.694 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librbd1. 2024-03-23T20:51:13.720 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../009-librbd1_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:13.761 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librbd1 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:13.831 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-23T20:51:13.857 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-23T20:51:13.897 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-23T20:51:14.163 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libcephfs2. 2024-03-23T20:51:14.190 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:14.221 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libcephfs2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:14.243 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-23T20:51:14.269 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-23T20:51:14.309 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-23T20:51:14.474 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rados. 2024-03-23T20:51:14.489 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../011-python3-rados_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:14.532 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rados (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:14.787 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-23T20:51:14.813 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-23T20:51:14.853 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-23T20:51:14.854 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-23T20:51:14.880 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:14.920 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-ceph-argparse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:15.155 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cephfs. 2024-03-23T20:51:15.171 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:15.205 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cephfs (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:15.324 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-23T20:51:15.350 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-23T20:51:15.399 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:51:15.450 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-23T20:51:15.465 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:15.502 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-ceph-common (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:15.769 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-23T20:51:15.795 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-23T20:51:15.827 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:51:15.961 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-23T20:51:15.988 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-23T20:51:16.028 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-23T20:51:16.222 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-23T20:51:16.249 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-23T20:51:16.288 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:51:16.347 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-prettytable. 2024-03-23T20:51:16.370 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-03-23T20:51:16.414 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-23T20:51:16.642 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-23T20:51:16.658 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rbd. 2024-03-23T20:51:16.668 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-23T20:51:16.684 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:16.708 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-23T20:51:16.724 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rbd (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:17.003 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librados2. 2024-03-23T20:51:17.010 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-23T20:51:17.029 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../008-librados2_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:17.033 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-23T20:51:17.069 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-23T20:51:17.069 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librados2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:17.365 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librgw2. 2024-03-23T20:51:17.390 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../019-librgw2_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:17.430 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librgw2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:17.472 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librbd1. 2024-03-23T20:51:17.499 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../009-librbd1_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:17.539 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librbd1 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:17.924 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rgw. 2024-03-23T20:51:17.950 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:17.984 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs2. 2024-03-23T20:51:17.990 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rgw (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:18.010 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:18.050 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:18.311 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rados. 2024-03-23T20:51:18.334 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../011-python3-rados_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:18.335 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-23T20:51:18.361 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-23T20:51:18.369 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rados (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:18.402 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-23T20:51:18.646 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libradosstriper1. 2024-03-23T20:51:18.672 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:18.680 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-23T20:51:18.704 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libradosstriper1 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:18.706 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:18.747 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-argparse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:19.008 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cephfs. 2024-03-23T20:51:19.024 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:19.066 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cephfs (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:19.116 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-common. 2024-03-23T20:51:19.142 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../023-ceph-common_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:19.182 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-common (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:19.411 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-23T20:51:19.438 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:19.477 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-common (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:19.780 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-23T20:51:19.807 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-23T20:51:19.947 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-23T20:51:20.292 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-prettytable. 2024-03-23T20:51:20.318 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-03-23T20:51:20.358 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-23T20:51:20.463 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-base. 2024-03-23T20:51:20.489 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../024-ceph-base_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:20.539 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-base (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:20.620 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rbd. 2024-03-23T20:51:20.635 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:20.678 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rbd (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:21.031 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-23T20:51:21.057 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-23T20:51:21.097 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-23T20:51:21.334 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-23T20:51:21.360 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-23T20:51:21.392 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librgw2. 2024-03-23T20:51:21.400 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-23T20:51:21.418 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../019-librgw2_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:21.450 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librgw2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:21.737 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cheroot. 2024-03-23T20:51:21.763 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-23T20:51:21.803 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-23T20:51:21.953 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rgw. 2024-03-23T20:51:21.979 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:22.019 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rgw (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:22.114 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-23T20:51:22.141 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-23T20:51:22.181 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-23T20:51:22.356 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-23T20:51:22.382 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-23T20:51:22.422 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-23T20:51:22.442 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-23T20:51:22.468 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-23T20:51:22.500 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-23T20:51:22.742 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libradosstriper1. 2024-03-23T20:51:22.768 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:22.778 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-23T20:51:22.806 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-23T20:51:22.808 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libradosstriper1 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:22.845 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-23T20:51:23.147 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-tempora. 2024-03-23T20:51:23.174 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-03-23T20:51:23.187 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-common. 2024-03-23T20:51:23.213 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../023-ceph-common_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:23.222 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-23T20:51:23.253 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-common (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:23.492 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-portend. 2024-03-23T20:51:23.518 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-03-23T20:51:23.558 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-23T20:51:23.828 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-23T20:51:23.880 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-23T20:51:23.894 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-23T20:51:24.181 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-23T20:51:24.207 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-23T20:51:24.248 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-23T20:51:24.501 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-base. 2024-03-23T20:51:24.527 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../024-ceph-base_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:24.559 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-natsort. 2024-03-23T20:51:24.574 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-03-23T20:51:24.576 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-base (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:24.608 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-23T20:51:24.920 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-logutils. 2024-03-23T20:51:24.947 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-03-23T20:51:24.987 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-23T20:51:25.289 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-23T20:51:25.305 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-23T20:51:25.348 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-23T20:51:25.405 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-23T20:51:25.431 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-23T20:51:25.463 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-23T20:51:25.642 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-mako. 2024-03-23T20:51:25.669 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-23T20:51:25.709 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-23T20:51:25.741 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cheroot. 2024-03-23T20:51:25.767 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-23T20:51:25.808 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-23T20:51:26.144 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-23T20:51:26.171 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-23T20:51:26.211 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-23T20:51:26.229 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-23T20:51:26.255 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-23T20:51:26.287 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-23T20:51:26.513 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-23T20:51:26.540 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-23T20:51:26.556 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-23T20:51:26.580 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-23T20:51:26.583 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-23T20:51:26.623 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-23T20:51:26.842 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-23T20:51:26.869 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-23T20:51:26.900 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-23T20:51:26.925 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webob. 2024-03-23T20:51:26.952 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-23T20:51:26.992 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-23T20:51:27.203 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-tempora. 2024-03-23T20:51:27.230 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-03-23T20:51:27.269 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-23T20:51:27.337 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-waitress. 2024-03-23T20:51:27.364 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-23T20:51:27.407 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-23T20:51:27.572 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-portend. 2024-03-23T20:51:27.599 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-03-23T20:51:27.639 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-23T20:51:27.681 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-tempita. 2024-03-23T20:51:27.708 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-23T20:51:27.740 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-23T20:51:27.917 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-23T20:51:27.943 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-23T20:51:27.983 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-23T20:51:28.008 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-paste. 2024-03-23T20:51:28.024 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-23T20:51:28.067 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-23T20:51:28.253 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-23T20:51:28.279 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-23T20:51:28.320 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-23T20:51:28.387 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-23T20:51:28.410 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-23T20:51:28.453 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-23T20:51:28.681 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-natsort. 2024-03-23T20:51:28.708 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-03-23T20:51:28.723 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-23T20:51:28.738 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-23T20:51:28.748 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-23T20:51:28.773 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-23T20:51:29.001 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-logutils. 2024-03-23T20:51:29.016 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-03-23T20:51:29.043 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webtest. 2024-03-23T20:51:29.052 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-23T20:51:29.069 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-03-23T20:51:29.109 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-23T20:51:29.346 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-23T20:51:29.371 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-23T20:51:29.412 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-23T20:51:29.437 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pecan. 2024-03-23T20:51:29.463 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-23T20:51:29.503 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-23T20:51:29.723 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-mako. 2024-03-23T20:51:29.750 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-23T20:51:29.790 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-23T20:51:29.866 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-certifi. 2024-03-23T20:51:29.893 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-03-23T20:51:29.933 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-23T20:51:30.077 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-23T20:51:30.104 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-23T20:51:30.144 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-23T20:51:30.269 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-idna. 2024-03-23T20:51:30.296 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2024-03-23T20:51:30.336 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-23T20:51:30.446 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-23T20:51:30.473 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-23T20:51:30.512 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-23T20:51:30.598 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-urllib3. 2024-03-23T20:51:30.624 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-23T20:51:30.655 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-23T20:51:30.808 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-webob. 2024-03-23T20:51:30.834 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-23T20:51:30.873 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-23T20:51:30.933 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-requests. 2024-03-23T20:51:30.960 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-23T20:51:30.991 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-23T20:51:31.243 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-waitress. 2024-03-23T20:51:31.270 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-23T20:51:31.313 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-23T20:51:31.419 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-23T20:51:31.447 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-23T20:51:31.486 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-23T20:51:31.571 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-tempita. 2024-03-23T20:51:31.586 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-23T20:51:31.621 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-23T20:51:31.722 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-23T20:51:31.738 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:31.772 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-modules-core (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:31.924 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-paste. 2024-03-23T20:51:31.951 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-23T20:51:31.990 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-23T20:51:32.144 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-23T20:51:32.170 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:32.210 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:32.294 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-23T20:51:32.320 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-23T20:51:32.352 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-23T20:51:32.504 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-openssl. 2024-03-23T20:51:32.531 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2024-03-23T20:51:32.571 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-23T20:51:32.680 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-23T20:51:32.706 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-23T20:51:32.746 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-23T20:51:32.874 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr. 2024-03-23T20:51:32.900 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../056-ceph-mgr_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:32.940 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:33.049 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-webtest. 2024-03-23T20:51:33.076 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-03-23T20:51:33.116 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-23T20:51:33.302 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mon. 2024-03-23T20:51:33.329 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../057-ceph-mon_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:33.368 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mon (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:33.427 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pecan. 2024-03-23T20:51:33.454 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-23T20:51:33.494 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-23T20:51:33.857 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-certifi. 2024-03-23T20:51:33.884 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-03-23T20:51:33.924 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-23T20:51:33.997 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-osd. 2024-03-23T20:51:34.024 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../058-ceph-osd_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:34.063 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-osd (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:34.210 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-idna. 2024-03-23T20:51:34.236 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2024-03-23T20:51:34.277 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-23T20:51:34.596 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-urllib3. 2024-03-23T20:51:34.623 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-23T20:51:34.662 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-23T20:51:34.983 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-requests. 2024-03-23T20:51:35.009 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-23T20:51:35.049 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-23T20:51:35.074 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph. 2024-03-23T20:51:35.090 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../059-ceph_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:35.124 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:36.178 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-fuse. 2024-03-23T20:51:36.206 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../060-ceph-fuse_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:36.287 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-fuse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:36.353 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-23T20:51:36.380 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-23T20:51:36.419 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-23T20:51:36.722 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-23T20:51:36.738 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:36.739 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mds. 2024-03-23T20:51:36.766 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../061-ceph-mds_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:36.781 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-modules-core (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:36.806 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mds (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:37.186 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-23T20:51:37.192 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package cephadm. 2024-03-23T20:51:37.213 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:37.219 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../062-cephadm_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:37.252 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:37.258 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking cephadm (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:37.497 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-openssl. 2024-03-23T20:51:37.524 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2024-03-23T20:51:37.545 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-23T20:51:37.555 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-23T20:51:37.572 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-23T20:51:37.603 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-23T20:51:37.889 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-23T20:51:37.891 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr. 2024-03-23T20:51:37.916 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-23T20:51:37.918 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../056-ceph-mgr_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:37.955 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-23T20:51:37.957 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:38.361 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mon. 2024-03-23T20:51:38.388 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../057-ceph-mon_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:38.428 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mon (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:38.919 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-babel. 2024-03-23T20:51:38.935 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-23T20:51:38.969 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-23T20:51:38.990 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-osd. 2024-03-23T20:51:39.016 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../058-ceph-osd_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:39.048 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-osd (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:39.264 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jinja2. 2024-03-23T20:51:39.291 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-23T20:51:39.331 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-23T20:51:39.633 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-23T20:51:39.649 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:39.683 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-cephadm (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:40.003 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-23T20:51:40.031 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-23T20:51:40.070 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-23T20:51:40.126 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph. 2024-03-23T20:51:40.142 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../059-ceph_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:40.184 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:40.372 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pluggy. 2024-03-23T20:51:40.399 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-23T20:51:40.439 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-23T20:51:40.504 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-fuse. 2024-03-23T20:51:40.531 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../060-ceph-fuse_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:40.570 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-fuse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:40.750 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-py. 2024-03-23T20:51:40.778 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-03-23T20:51:40.818 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-23T20:51:40.907 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mds. 2024-03-23T20:51:40.933 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../061-ceph-mds_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:40.965 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mds (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:41.095 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-23T20:51:41.111 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-23T20:51:41.153 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-23T20:51:41.368 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cephadm. 2024-03-23T20:51:41.395 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../062-cephadm_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:41.431 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-toml. 2024-03-23T20:51:41.435 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cephadm (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:41.459 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-03-23T20:51:41.498 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-23T20:51:41.755 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-23T20:51:41.781 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-23T20:51:41.784 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pytest. 2024-03-23T20:51:41.811 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-23T20:51:41.821 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-23T20:51:41.850 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-23T20:51:42.149 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-23T20:51:42.176 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-23T20:51:42.215 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-23T20:51:42.221 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-23T20:51:42.249 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-03-23T20:51:42.287 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-23T20:51:42.548 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-routes. 2024-03-23T20:51:42.575 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-23T20:51:42.606 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-23T20:51:42.902 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-23T20:51:42.917 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:42.952 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-dashboard (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:43.221 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-babel. 2024-03-23T20:51:43.236 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-23T20:51:43.270 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-23T20:51:43.363 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-23T20:51:43.379 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-23T20:51:43.421 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-23T20:51:43.574 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jinja2. 2024-03-23T20:51:43.600 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-23T20:51:43.640 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-23T20:51:43.892 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-joblib. 2024-03-23T20:51:43.919 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-23T20:51:43.958 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-23T20:51:43.985 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-23T20:51:44.012 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:44.052 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-cephadm (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:44.279 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-23T20:51:44.306 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-23T20:51:44.336 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-23T20:51:44.372 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-23T20:51:44.399 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-23T20:51:44.438 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-23T20:51:44.631 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-sklearn. 2024-03-23T20:51:44.658 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-23T20:51:44.697 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-23T20:51:44.741 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pluggy. 2024-03-23T20:51:44.768 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-23T20:51:44.808 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-23T20:51:45.120 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-py. 2024-03-23T20:51:45.146 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-03-23T20:51:45.186 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-23T20:51:45.356 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-23T20:51:45.384 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:45.422 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:45.490 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-23T20:51:45.517 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-23T20:51:45.556 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-23T20:51:45.826 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-toml. 2024-03-23T20:51:45.852 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-03-23T20:51:45.892 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-23T20:51:46.358 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cachetools. 2024-03-23T20:51:46.381 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pytest. 2024-03-23T20:51:46.386 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-03-23T20:51:46.408 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-23T20:51:46.425 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-23T20:51:46.464 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-23T20:51:46.801 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-23T20:51:46.817 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-03-23T20:51:46.859 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-23T20:51:46.996 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-23T20:51:47.023 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-23T20:51:47.062 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-23T20:51:47.171 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-routes. 2024-03-23T20:51:47.198 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-23T20:51:47.237 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-23T20:51:47.399 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-23T20:51:47.426 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-23T20:51:47.466 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-23T20:51:47.583 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-23T20:51:47.609 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:47.649 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-dashboard (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:47.761 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rsa. 2024-03-23T20:51:47.776 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-03-23T20:51:47.810 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-23T20:51:48.086 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-23T20:51:48.101 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-23T20:51:48.114 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-google-auth. 2024-03-23T20:51:48.141 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-03-23T20:51:48.144 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-23T20:51:48.180 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-23T20:51:48.484 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-23T20:51:48.511 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-23T20:51:48.550 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-23T20:51:48.640 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-joblib. 2024-03-23T20:51:48.667 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-23T20:51:48.706 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-23T20:51:48.837 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-websocket. 2024-03-23T20:51:48.864 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-03-23T20:51:48.895 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-23T20:51:49.085 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-23T20:51:49.113 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-23T20:51:49.152 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-23T20:51:49.173 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-23T20:51:49.189 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-23T20:51:49.223 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-23T20:51:49.480 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-sklearn. 2024-03-23T20:51:49.507 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-23T20:51:49.546 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-23T20:51:49.853 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-23T20:51:49.869 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:49.911 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:50.181 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-23T20:51:50.208 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-23T20:51:50.239 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-23T20:51:50.289 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-23T20:51:50.316 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:50.355 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:50.592 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-23T20:51:50.619 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-23T20:51:50.658 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-23T20:51:50.928 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package jq. 2024-03-23T20:51:50.943 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-23T20:51:50.986 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-23T20:51:51.214 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package socat. 2024-03-23T20:51:51.241 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-23T20:51:51.280 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-23T20:51:51.283 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cachetools. 2024-03-23T20:51:51.299 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-03-23T20:51:51.333 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-23T20:51:51.558 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package xmlstarlet. 2024-03-23T20:51:51.574 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-23T20:51:51.608 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-23T20:51:51.713 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-23T20:51:51.740 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-23T20:51:51.779 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-23T20:51:51.937 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-test. 2024-03-23T20:51:51.953 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../096-ceph-test_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:51.987 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-test (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:52.108 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-23T20:51:52.135 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-23T20:51:52.174 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-23T20:51:52.478 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rsa. 2024-03-23T20:51:52.505 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-03-23T20:51:52.544 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-23T20:51:52.832 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-google-auth. 2024-03-23T20:51:52.859 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-03-23T20:51:52.898 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-23T20:51:53.210 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-23T20:51:53.237 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-23T20:51:53.276 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-23T20:51:53.596 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-websocket. 2024-03-23T20:51:53.623 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-03-23T20:51:53.663 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-23T20:51:53.799 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-volume. 2024-03-23T20:51:53.815 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:53.849 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-volume (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:53.974 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-23T20:51:54.002 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-23T20:51:54.041 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-23T20:51:54.113 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-23T20:51:54.129 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:54.172 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libcephfs-dev (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:54.558 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package nvme-cli. 2024-03-23T20:51:54.585 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-23T20:51:54.624 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-23T20:51:54.730 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-23T20:51:54.758 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:54.797 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:54.971 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-23T20:51:54.987 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-23T20:51:55.029 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-23T20:51:55.116 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-23T20:51:55.143 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-23T20:51:55.183 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-23T20:51:55.457 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pastescript. 2024-03-23T20:51:55.485 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-03-23T20:51:55.494 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-23T20:51:55.510 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-23T20:51:55.523 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-23T20:51:55.544 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-23T20:51:55.814 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package jq. 2024-03-23T20:51:55.841 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-23T20:51:55.851 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-psutil. 2024-03-23T20:51:55.879 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-23T20:51:55.880 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-23T20:51:55.918 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-23T20:51:56.150 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package socat. 2024-03-23T20:51:56.177 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-23T20:51:56.216 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-23T20:51:56.238 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pygments. 2024-03-23T20:51:56.265 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-23T20:51:56.304 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-23T20:51:56.828 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package xmlstarlet. 2024-03-23T20:51:56.856 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-23T20:51:56.894 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-23T20:51:56.942 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-23T20:51:56.969 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-23T20:51:57.008 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-23T20:51:57.258 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-test. 2024-03-23T20:51:57.285 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../096-ceph-test_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:57.294 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-simplejson. 2024-03-23T20:51:57.321 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-23T20:51:57.332 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-test (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:57.360 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-23T20:51:57.613 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-23T20:51:57.629 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-23T20:51:57.671 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-23T20:51:58.417 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package radosgw. 2024-03-23T20:51:58.444 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../107-radosgw_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:58.483 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking radosgw (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:59.211 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package rbd-fuse. 2024-03-23T20:51:59.239 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:59.278 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking rbd-fuse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:59.295 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-volume. 2024-03-23T20:51:59.323 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-954-g82316b04-1jammy_all.deb ... 2024-03-23T20:51:59.361 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-volume (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:59.572 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package rbd-mirror. 2024-03-23T20:51:59.599 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../109-rbd-mirror_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:59.630 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking rbd-mirror (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:51:59.649 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-23T20:51:59.665 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:51:59.699 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs-dev (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:00.052 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package nvme-cli. 2024-03-23T20:52:00.067 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-23T20:52:00.080 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-23T20:52:00.118 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-23T20:52:00.318 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-23T20:52:00.444 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-23T20:52:00.449 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-23T20:52:00.476 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-23T20:52:00.515 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-23T20:52:00.709 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-23T20:52:00.944 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pastescript. 2024-03-23T20:52:00.971 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-03-23T20:52:01.010 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-23T20:52:01.022 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-23T20:52:01.330 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-psutil. 2024-03-23T20:52:01.358 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-23T20:52:01.397 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-23T20:52:01.435 INFO:teuthology.orchestra.run.smithi117.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-23T20:52:01.496 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up cephadm (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:01.742 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pygments. 2024-03-23T20:52:01.769 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-23T20:52:01.825 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-23T20:52:01.962 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user cephadm....done 2024-03-23T20:52:02.229 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-23T20:52:02.245 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-23T20:52:02.252 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-23T20:52:02.287 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-23T20:52:02.615 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-simplejson. 2024-03-23T20:52:02.643 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-23T20:52:02.681 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-23T20:52:02.694 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-23T20:52:02.946 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-23T20:52:02.950 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-23T20:52:02.967 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-23T20:52:03.009 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-23T20:52:03.071 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-23T20:52:03.322 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-23T20:52:03.574 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-23T20:52:03.699 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-23T20:52:03.763 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package radosgw. 2024-03-23T20:52:03.791 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../107-radosgw_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:52:03.838 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking radosgw (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:04.001 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-23T20:52:04.361 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-23T20:52:04.566 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package rbd-fuse. 2024-03-23T20:52:04.582 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:52:04.616 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking rbd-fuse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:04.620 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-23T20:52:04.888 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-23T20:52:04.936 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package rbd-mirror. 2024-03-23T20:52:04.963 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../109-rbd-mirror_18.2.2-954-g82316b04-1jammy_amd64.deb ... 2024-03-23T20:52:05.002 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking rbd-mirror (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:05.139 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-23T20:52:05.390 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-23T20:52:05.465 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-23T20:52:05.641 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-ceph-argparse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:05.716 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-23T20:52:05.817 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-23T20:52:05.900 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-23T20:52:06.080 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-23T20:52:06.160 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-23T20:52:06.386 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-23T20:52:06.419 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-23T20:52:06.545 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-23T20:52:06.798 INFO:teuthology.orchestra.run.smithi204.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-23T20:52:06.898 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cephadm (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:06.904 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-23T20:52:07.022 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-23T20:52:07.290 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-23T20:52:07.641 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-23T20:52:07.736 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user cephadm....done 2024-03-23T20:52:07.935 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-23T20:52:07.942 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-23T20:52:08.428 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-23T20:52:08.603 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-23T20:52:08.688 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-23T20:52:08.814 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-23T20:52:08.853 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-23T20:52:08.980 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-23T20:52:09.074 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-23T20:52:09.106 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-23T20:52:09.317 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-23T20:52:09.407 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-23T20:52:09.417 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-23T20:52:09.744 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-23T20:52:09.791 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-23T20:52:10.034 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-23T20:52:10.130 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-23T20:52:10.381 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-23T20:52:10.402 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-23T20:52:10.649 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-23T20:52:10.703 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-23T20:52:10.891 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-23T20:52:11.038 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-23T20:52:11.134 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-23T20:52:11.289 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-23T20:52:11.394 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-argparse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:11.415 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-ceph-common (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:11.653 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-23T20:52:11.708 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-idna (3.3-1) ... 2024-03-23T20:52:11.780 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-23T20:52:12.009 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-23T20:52:12.031 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-23T20:52:12.157 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-23T20:52:12.500 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-23T20:52:12.626 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-23T20:52:12.869 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-23T20:52:13.128 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-23T20:52:13.181 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:52:13.315 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-23T20:52:13.463 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-23T20:52:13.575 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-23T20:52:13.701 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-23T20:52:13.827 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-23T20:52:14.107 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-23T20:52:14.128 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-23T20:52:14.358 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-23T20:52:14.387 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-23T20:52:14.484 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-23T20:52:14.610 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-23T20:52:14.638 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:52:14.764 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-03-23T20:52:14.890 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-23T20:52:14.911 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-23T20:52:15.200 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-23T20:52:15.296 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-23T20:52:15.451 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-23T20:52:15.556 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-23T20:52:15.710 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-23T20:52:15.941 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-23T20:52:15.970 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-23T20:52:16.204 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-23T20:52:16.208 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-23T20:52:16.455 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-23T20:52:16.568 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-23T20:52:16.581 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-23T20:52:16.819 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-23T20:52:16.865 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-23T20:52:16.945 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-common (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:16.992 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-23T20:52:17.258 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-idna (3.3-1) ... 2024-03-23T20:52:17.276 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-23T20:52:17.577 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-23T20:52:17.623 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-23T20:52:17.812 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-23T20:52:18.174 INFO:teuthology.orchestra.run.smithi117.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-23T20:52:18.415 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-23T20:52:18.541 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-23T20:52:18.771 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:52:18.914 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-23T20:52:18.917 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-23T20:52:19.165 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-23T20:52:19.201 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-23T20:52:19.291 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-23T20:52:19.392 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-23T20:52:19.461 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:52:19.578 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-23T20:52:19.677 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-23T20:52:19.871 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-23T20:52:19.928 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-23T20:52:20.179 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:52:20.180 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-23T20:52:20.297 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-03-23T20:52:20.406 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-23T20:52:20.457 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-23T20:52:20.582 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-23T20:52:20.700 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-23T20:52:20.850 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-23T20:52:20.942 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-23T20:52:21.118 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-23T20:52:21.218 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-23T20:52:21.377 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-23T20:52:21.470 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-23T20:52:21.721 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-23T20:52:21.770 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-23T20:52:21.989 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-23T20:52:22.063 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-23T20:52:22.115 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-23T20:52:22.383 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-23T20:52:22.399 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-23T20:52:22.508 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librados2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:22.509 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-23T20:52:22.634 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librgw2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:22.760 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libsqlite3-mod-ceph (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:22.827 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-23T20:52:23.086 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libcephfs2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:23.145 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-23T20:52:23.229 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libradosstriper1 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:23.404 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-23T20:52:23.440 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-23T20:52:23.708 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librbd1 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:23.834 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-modules-core (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:23.838 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-23T20:52:23.960 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-fuse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:24.082 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-23T20:52:24.175 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-23T20:52:24.175 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-23T20:52:24.208 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-23T20:52:24.602 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-23T20:52:24.661 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libcephfs-dev (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:24.787 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rados (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:24.861 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-23T20:52:24.913 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-23T20:52:25.113 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-23T20:52:25.247 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-23T20:52:25.565 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-23T20:52:25.841 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-23T20:52:26.134 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-23T20:52:26.142 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rbd (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:26.252 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up rbd-fuse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:26.260 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-23T20:52:26.361 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rgw (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:26.470 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cephfs (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:26.536 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-23T20:52:26.596 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-common (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:26.812 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-23T20:52:27.080 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-23T20:52:27.147 INFO:teuthology.orchestra.run.smithi117.stdout:Adding group ceph....done 2024-03-23T20:52:27.448 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-23T20:52:27.516 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user ceph....done 2024-03-23T20:52:27.684 INFO:teuthology.orchestra.run.smithi117.stdout:Setting system user ceph properties....done 2024-03-23T20:52:27.699 INFO:teuthology.orchestra.run.smithi117.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-23T20:52:27.835 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-23T20:52:27.858 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-23T20:52:28.171 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-23T20:52:28.193 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-23T20:52:28.320 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librados2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:28.446 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librgw2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:28.564 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libsqlite3-mod-ceph (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:28.630 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-test (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:28.673 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs2 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:28.748 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up radosgw (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:28.791 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libradosstriper1 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:29.011 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-23T20:52:29.181 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-23T20:52:29.182 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-23T20:52:29.287 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librbd1 (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:29.422 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-modules-core (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:29.531 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-fuse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:29.641 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-base (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:29.730 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-23T20:52:29.730 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-23T20:52:29.993 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-23T20:52:30.191 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs-dev (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:30.317 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rados (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:30.427 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mds (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:30.443 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-23T20:52:30.642 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-23T20:52:30.642 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-23T20:52:31.095 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up rbd-mirror (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:31.307 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T20:52:31.307 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T20:52:31.692 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rbd (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:31.780 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:31.817 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up rbd-fuse (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:31.943 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rgw (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:31.997 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-23T20:52:31.997 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-23T20:52:32.069 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cephfs (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:32.196 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-common (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:32.473 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-osd (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:32.729 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-23T20:52:32.730 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-23T20:52:32.764 INFO:teuthology.orchestra.run.smithi204.stdout:Adding group ceph....done 2024-03-23T20:52:33.148 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user ceph....done 2024-03-23T20:52:33.199 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-k8sevents (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:33.275 INFO:teuthology.orchestra.run.smithi204.stdout:Setting system user ceph properties....done 2024-03-23T20:52:33.292 INFO:teuthology.orchestra.run.smithi204.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-23T20:52:33.317 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:33.430 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-23T20:52:33.469 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mon (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:33.682 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-23T20:52:33.682 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-23T20:52:33.792 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-23T20:52:34.169 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-cephadm (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:34.247 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-test (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:34.312 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:34.373 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up radosgw (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:34.438 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-dashboard (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:34.580 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-volume (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:34.831 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-23T20:52:34.832 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-23T20:52:35.025 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-23T20:52:35.325 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-base (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:35.674 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-23T20:52:36.150 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-23T20:52:36.152 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mds (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:36.378 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-23T20:52:36.378 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-23T20:52:36.589 INFO:teuthology.orchestra.run.smithi117.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-23T20:52:36.595 DEBUG:teuthology.parallel:result is None 2024-03-23T20:52:36.854 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up rbd-mirror (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:37.057 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T20:52:37.058 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T20:52:37.547 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:37.748 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-23T20:52:37.748 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-23T20:52:38.240 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-osd (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:38.502 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-23T20:52:38.502 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-23T20:52:38.992 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-k8sevents (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:39.118 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:39.294 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mon (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:39.511 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-23T20:52:39.512 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-23T20:52:39.978 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-cephadm (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:40.130 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:40.247 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-dashboard (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:40.374 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-volume (18.2.2-954-g82316b04-1jammy) ... 2024-03-23T20:52:40.702 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-23T20:52:41.762 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-23T20:52:42.233 INFO:teuthology.orchestra.run.smithi204.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-23T20:52:42.238 DEBUG:teuthology.parallel:result is None 2024-03-23T20:52:42.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T20:52:42.380 DEBUG:teuthology.orchestra.run.smithi117:> dpkg-query -W -f '${Version}' ceph 2024-03-23T20:52:42.399 INFO:teuthology.orchestra.run.smithi117.stdout:18.2.2-954-g82316b04-1jammy 2024-03-23T20:52:42.400 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-954-g82316b04-1jammy 2024-03-23T20:52:42.400 INFO:teuthology.task.install:The correct ceph version 18.2.2-954-g82316b04-1jammy is installed. 2024-03-23T20:52:42.402 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T20:52:42.615 DEBUG:teuthology.orchestra.run.smithi204:> dpkg-query -W -f '${Version}' ceph 2024-03-23T20:52:42.633 INFO:teuthology.orchestra.run.smithi204.stdout:18.2.2-954-g82316b04-1jammy 2024-03-23T20:52:42.634 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-954-g82316b04-1jammy 2024-03-23T20:52:42.634 INFO:teuthology.task.install:The correct ceph version 18.2.2-954-g82316b04-1jammy is installed. 2024-03-23T20:52:42.635 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-23T20:52:42.636 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:42.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T20:52:42.650 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:42.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T20:52:42.695 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-23T20:52:42.695 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:42.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T20:52:42.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T20:52:42.765 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:42.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T20:52:42.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T20:52:42.833 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-23T20:52:42.834 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:42.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T20:52:42.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T20:52:42.900 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:42.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T20:52:42.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T20:52:42.970 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-23T20:52:42.970 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:42.970 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T20:52:42.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T20:52:43.036 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:43.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T20:52:43.049 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T20:52:43.105 INFO:teuthology.run_tasks:Running task ceph... 2024-03-23T20:52:43.172 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-23T20:52:43.173 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-03-23T20:52:43.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2024-03-23T20:52:43.188 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-23T20:52:43.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T20:52:43.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T20:52:43.243 INFO:tasks.ceph:Creating extra log directories... 2024-03-23T20:52:43.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T20:52:43.287 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T20:52:43.312 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-03-23T20:52:43.312 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-23T20:52:43.312 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668', 'branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7617668', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri11-testing-2024-03-21-0851-reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 6304, 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'targets': {'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1jcoioqpma6131vrwi23FPocv1RZFh/qwKKJR0unECZSCSz46OXgHJIhVax58gSn+DqTttcf+hwf4JLjyGxfs=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP3lLbZ167RBEDXN5OmvB8wahaNtp28lVj7Ctgps/XZan0ZhVsbikIRpq37YPGsjrb4IZAdK/KxIOad73zDNz7Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_journal.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-22_13:19:01', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-23T20:52:43.313 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-03-23T20:52:43.343 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-03-23T20:52:43.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T20:52:43.391 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T20:52:43.405 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:43.406 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T20:52:43.450 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T20:52:43.450 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-03-23T20:52:43.498 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T20:52:43.498 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T20:52:43.498 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-03-23T20:52:43.498 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T20:52:43.498 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-23 20:47:24.359514950 +0000 2024-03-23T20:52:43.498 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-23 20:47:24.111517165 +0000 2024-03-23T20:52:43.498 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-23 20:47:24.111517165 +0000 2024-03-23T20:52:43.498 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2024-03-23T20:52:43.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T20:52:43.552 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-23T20:52:43.552 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-23T20:52:43.552 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000447697 s, 1.1 MB/s 2024-03-23T20:52:43.554 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T20:52:43.604 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-03-23T20:52:43.650 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T20:52:43.650 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T20:52:43.650 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-03-23T20:52:43.650 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T20:52:43.650 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-23 20:47:24.859510484 +0000 2024-03-23T20:52:43.650 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-23 20:47:24.595512842 +0000 2024-03-23T20:52:43.650 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-23 20:47:24.595512842 +0000 2024-03-23T20:52:43.650 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2024-03-23T20:52:43.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T20:52:43.704 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-23T20:52:43.705 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-23T20:52:43.705 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000434947 s, 1.2 MB/s 2024-03-23T20:52:43.706 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T20:52:43.755 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-03-23T20:52:43.801 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T20:52:43.801 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T20:52:43.801 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-03-23T20:52:43.802 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T20:52:43.802 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-23 20:47:25.375505875 +0000 2024-03-23T20:52:43.802 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-23 20:47:25.103508304 +0000 2024-03-23T20:52:43.802 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-23 20:47:25.103508304 +0000 2024-03-23T20:52:43.802 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2024-03-23T20:52:43.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T20:52:43.856 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-23T20:52:43.856 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-23T20:52:43.856 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000426194 s, 1.2 MB/s 2024-03-23T20:52:43.857 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T20:52:43.908 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-03-23T20:52:43.954 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T20:52:43.954 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T20:52:43.954 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 940 Links: 1 2024-03-23T20:52:43.954 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T20:52:43.954 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-23 20:47:25.931500909 +0000 2024-03-23T20:52:43.954 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-23 20:47:25.639503517 +0000 2024-03-23T20:52:43.954 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-23 20:47:25.639503517 +0000 2024-03-23T20:52:43.954 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2024-03-23T20:52:43.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T20:52:44.008 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-23T20:52:44.008 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-23T20:52:44.008 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000387354 s, 1.3 MB/s 2024-03-23T20:52:44.010 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T20:52:44.059 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-23T20:52:44.059 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-23T20:52:44.059 INFO:tasks.ceph:Generating config... 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client] debug rbd = 20 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client] log to stderr = False 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client] rbd default features = 125 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.061 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.062 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.062 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.062 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.062 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.062 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.062 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-23T20:52:44.062 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-03-23T20:52:44.062 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-23T20:52:44.062 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-23T20:52:44.062 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-23T20:52:44.062 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-23T20:52:44.062 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-23T20:52:44.062 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-23T20:52:44.063 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-23T20:52:44.064 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-23T20:52:44.064 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-03-23T20:52:44.064 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-03-23T20:52:44.131 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/cluster1.keyring 2024-03-23T20:52:44.146 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-03-23T20:52:44.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-03-23T20:52:44.277 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.117')] 2024-03-23T20:52:44.277 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.117', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-03-23T20:52:44.278 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:44.278 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T20:52:44.326 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.117 --print /home/ubuntu/cephtest/cluster1.monmap 2024-03-23T20:52:44.403 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: generated fsid f60222f0-44b2-480a-bdc8-0eb22ebe7b33 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:setting min_mon_release = pacific 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:epoch 0 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:fsid f60222f0-44b2-480a-bdc8-0eb22ebe7b33 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:last_changed 2024-03-23T20:52:44.406301+0000 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:created 2024-03-23T20:52:44.406301+0000 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:min_mon_release 16 (pacific) 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:election_strategy: 1 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-23T20:52:44.404 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-03-23T20:52:44.405 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T20:52:44.454 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID f60222f0-44b2-480a-bdc8-0eb22ebe7b33... 2024-03-23T20:52:44.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-03-23T20:52:44.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-03-23T20:52:44.528 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-03-23T20:52:44.528 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-03-23T20:52:44.528 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T20:52:44.528 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-03-23T20:52:44.528 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.529 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-03-23T20:52:44.529 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.529 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-03-23T20:52:44.529 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.529 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-03-23T20:52:44.529 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-03-23T20:52:44.530 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.530 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-03-23T20:52:44.530 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-03-23T20:52:44.530 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.530 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T20:52:44.530 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-03-23T20:52:44.531 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.531 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T20:52:44.531 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T20:52:44.531 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.531 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-03-23T20:52:44.531 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-03-23T20:52:44.532 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = 172.21.15.117 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: mon client directed command retry = 5 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on pool no app = False 2024-03-23T20:52:44.533 INFO:teuthology.orchestra.run.smithi117.stdout: ms inject socket failures = 5000 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = f60222f0-44b2-480a-bdc8-0eb22ebe7b33 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-03-23T20:52:44.534 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.535 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: bdev async discard = True 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: bdev enable discard = True 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore allocator = stupid 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 1/20 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 1/20 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 4/10 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-03-23T20:52:44.536 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-23T20:52:44.537 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.538 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T20:52:44.539 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T20:52:44.539 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T20:52:44.539 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.539 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T20:52:44.540 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T20:52:44.540 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T20:52:44.540 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.540 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-03-23T20:52:44.540 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: debug rbd = 20 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: debug rbd_mirror = 15 2024-03-23T20:52:44.541 INFO:teuthology.orchestra.run.smithi117.stdout: log to stderr = False 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: rbd default features = 125 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.0] 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.1] 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.2] 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.3] 2024-03-23T20:52:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.4] 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.5] 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.6] 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.543 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.545 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-03-23T20:52:44.546 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-03-23T20:52:44.547 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = 172.21.15.117 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on pool no app = False 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = f60222f0-44b2-480a-bdc8-0eb22ebe7b33 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.548 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-03-23T20:52:44.549 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-03-23T20:52:44.549 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-03-23T20:52:44.549 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.550 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: bdev async discard = True 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: bdev enable discard = True 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore allocator = stupid 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2024-03-23T20:52:44.551 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 1/20 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 1/20 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 4/10 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-23T20:52:44.552 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-03-23T20:52:44.553 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T20:52:44.554 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-03-23T20:52:44.555 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd = 20 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd_mirror = 15 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: log to stderr = False 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout: rbd default features = 125 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-03-23T20:52:44.556 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.4] 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.557 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.558 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.5] 2024-03-23T20:52:44.558 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.558 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.558 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.6] 2024-03-23T20:52:44.558 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:52:44.558 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:52:44.558 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-03-23T20:52:44.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-03-23T20:52:44.637 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-23T20:52:44.637 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:44.637 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-03-23T20:52:44.643 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:44.643 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-03-23T20:52:44.690 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T20:52:44.690 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:44.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/cluster1.keyring 2024-03-23T20:52:44.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-03-23T20:52:44.754 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:44.754 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-03-23T20:52:44.802 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T20:52:44.802 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:44.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/cluster1.keyring 2024-03-23T20:52:44.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-03-23T20:52:44.826 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:44.826 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-03-23T20:52:44.874 INFO:tasks.ceph:Setting up mon nodes... 2024-03-23T20:52:44.875 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-23T20:52:44.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-03-23T20:52:44.917 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-03-23T20:52:44.920 INFO:tasks.ceph:Setting up mds nodes... 2024-03-23T20:52:44.920 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-23T20:52:44.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-03-23T20:52:44.993 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-03-23T20:52:45.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-03-23T20:52:45.039 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-03-23T20:52:45.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-03-23T20:52:45.139 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-03-23T20:52:45.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-03-23T20:52:45.222 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-03-23T20:52:45.231 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-03-23T20:52:45.301 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-03-23T20:52:45.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-03-23T20:52:45.383 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-03-23T20:52:45.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-03-23T20:52:45.462 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-03-23T20:52:45.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-03-23T20:52:45.543 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-03-23T20:52:45.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-03-23T20:52:45.627 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-03-23T20:52:45.636 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-23T20:52:45.636 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-23T20:52:45.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-03-23T20:52:45.651 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-23T20:52:45.651 INFO:tasks.ceph:role: cluster1.osd.0 2024-03-23T20:52:45.651 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T20:52:45.651 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T20:52:45.714 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T20:52:46.165 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-23T20:52:46.166 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-23T20:52:46.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-03-23T20:52:46.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-03-23T20:52:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:52:46.267 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2024-03-23T20:52:46.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-03-23T20:52:46.320 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-23T20:52:46.320 INFO:tasks.ceph:role: cluster1.osd.1 2024-03-23T20:52:46.320 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T20:52:46.321 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T20:52:46.380 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T20:52:46.847 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-23T20:52:46.848 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-23T20:52:46.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-03-23T20:52:46.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-03-23T20:52:46.998 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2024-03-23T20:52:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:52:46.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-03-23T20:52:47.054 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-23T20:52:47.054 INFO:tasks.ceph:role: cluster1.osd.2 2024-03-23T20:52:47.054 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T20:52:47.054 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-23T20:52:47.118 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T20:52:47.119 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T20:52:47.607 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-23T20:52:47.609 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-23T20:52:47.610 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-03-23T20:52:47.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-03-23T20:52:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:52:47.772 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2024-03-23T20:52:47.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-03-23T20:52:47.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:47.886+0000 7fc61c8f0640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-03-23T20:52:47.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:47.886+0000 7fc61c8f0640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-03-23T20:52:47.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:47.886+0000 7fc61c8f0640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-03-23T20:52:47.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:47.886+0000 7fc61c8f0640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-03-23T20:52:47.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:47.886+0000 7fc61c8f0640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-03-23T20:52:47.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:47.886+0000 7fc61c8f0640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-03-23T20:52:50.866 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-03-23T20:52:50.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-03-23T20:52:50.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:50.962+0000 7f142d1f3640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-03-23T20:52:50.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:50.962+0000 7f142d1f3640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-03-23T20:52:50.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:50.962+0000 7f142d1f3640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-03-23T20:52:50.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:50.962+0000 7f142d1f3640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-03-23T20:52:50.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:50.962+0000 7f142d1f3640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-03-23T20:52:50.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:50.966+0000 7f142d1f3640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-03-23T20:52:53.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-03-23T20:52:53.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-03-23T20:52:54.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:54.074+0000 7fd22c514640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-03-23T20:52:54.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:54.074+0000 7fd22c514640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-03-23T20:52:54.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:54.074+0000 7fd22c514640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-03-23T20:52:54.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:54.074+0000 7fd22c514640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-03-23T20:52:54.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:54.074+0000 7fd22c514640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-03-23T20:52:54.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-23T20:52:54.078+0000 7fd22c514640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-03-23T20:52:57.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-03-23T20:52:57.106 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-23T20:52:57.106 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:57.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-03-23T20:52:57.161 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:57.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-03-23T20:52:57.218 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:57.218 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-03-23T20:52:57.273 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:57.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-03-23T20:52:57.329 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:57.330 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-03-23T20:52:57.378 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:57.378 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-03-23T20:52:57.384 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:57.385 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-03-23T20:52:57.430 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:57.430 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-03-23T20:52:57.478 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:57.478 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-03-23T20:52:57.526 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:57.527 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-03-23T20:52:57.574 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:57.575 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-03-23T20:52:57.622 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:57.622 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-03-23T20:52:57.670 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:52:57.671 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-03-23T20:52:57.718 INFO:tasks.ceph:Adding keys to all mons... 2024-03-23T20:52:57.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/cluster1.keyring 2024-03-23T20:52:57.731 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAcQf9lHUnZNhAAVwYLgODBuMW+bObvt0IGOQ== 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAfQf9lMOb2NBAAWmhqHlM2QVeQDk3iRySt7w== 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAiQf9lKkmnORAA1KNxwOctaozGAJ/rybVD7A== 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAmQf9lVgmyBBAAkg1ZLnsPSbsRiiRrq5XhzQ== 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAcQf9l/wJfOxAAzwmFWxrhRfLgIpp97d/a9w== 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror] 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAdQf9lFdduAhAADu7AmwmuPZc/gCdu4Cqceg== 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.0] 2024-03-23T20:52:57.732 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAdQf9l4a1eCBAAMvqhSMUw7nL2+Y6o+2+Y1g== 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.1] 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAdQf9lf15WDRAAGJTh9og02SlremBBSKq5sQ== 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.2] 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAdQf9lkhQNEhAADgL8/pWKugnNq6WJC9Bnhg== 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.3] 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAdQf9lPfrwFhAAdleXP5n1IQ0IQH0FMYWbcg== 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.4] 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAdQf9lrYylGxAAlC8Sk2n+mGY3BwVbqg5SFg== 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.5] 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAdQf9lfnx4IBAA8oeaeJ9HWCoFj8mff2DLJg== 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.6] 2024-03-23T20:52:57.733 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAdQf9lkNl5JRAAZiXnQCAIifMsuiWlLpBidg== 2024-03-23T20:52:57.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T20:52:57.821 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T20:52:57.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T20:52:57.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T20:52:58.072 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.165 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.641 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:52:58.771 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-23T20:52:58.771 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-03-23T20:52:58.781 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-03-23T20:52:58.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-03-23T20:52:58.923 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-03-23T20:52:58.973 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-03-23T20:52:58.974 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-03-23T20:52:58.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-03-23T20:52:59.019 INFO:tasks.ceph.cluster1.mon.a:Started 2024-03-23T20:52:59.019 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-03-23T20:52:59.019 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-03-23T20:52:59.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-03-23T20:52:59.021 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-03-23T20:52:59.022 INFO:tasks.ceph:Setting crush tunables to default 2024-03-23T20:52:59.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster cluster1 osd crush tunables default 2024-03-23T20:52:59.252 INFO:teuthology.orchestra.run.smithi117.stderr:adjusted tunables profile to default 2024-03-23T20:52:59.269 INFO:tasks.ceph:check_enable_crimson: False 2024-03-23T20:52:59.269 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-03-23T20:52:59.270 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:59.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-03-23T20:52:59.279 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:59.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-03-23T20:52:59.330 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:52:59.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-03-23T20:52:59.359 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:52:59.358+0000 7f2c0df5c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T20:52:59.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster cluster1 osd new f0363173-1820-4027-8318-a96cb3f0e5f0 0 2024-03-23T20:52:59.612 INFO:teuthology.orchestra.run.smithi117.stdout:0 2024-03-23T20:52:59.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster cluster1 osd new c6028d94-1be7-43bb-8c5c-999d150aa717 1 2024-03-23T20:52:59.826 INFO:teuthology.orchestra.run.smithi117.stdout:1 2024-03-23T20:52:59.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster cluster1 osd new 8595a61f-9a3a-4503-9214-1f7dd7d385a0 2 2024-03-23T20:52:59.856 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:52:59.854+0000 7f2c0df5c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T20:53:00.036 INFO:teuthology.orchestra.run.smithi117.stdout:2 2024-03-23T20:53:00.053 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-03-23T20:53:00.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-03-23T20:53:00.077 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:00.078+0000 7f2c0df5c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T20:53:00.079 INFO:tasks.ceph.cluster1.osd.0:Started 2024-03-23T20:53:00.079 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-03-23T20:53:00.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-03-23T20:53:00.081 INFO:tasks.ceph.cluster1.osd.1:Started 2024-03-23T20:53:00.081 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-03-23T20:53:00.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-03-23T20:53:00.083 INFO:tasks.ceph.cluster1.osd.2:Started 2024-03-23T20:53:00.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-03-23T20:53:00.269 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:00.269 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"fsid":"f60222f0-44b2-480a-bdc8-0eb22ebe7b33","created":"2024-03-23T20:52:59.149423+0000","modified":"2024-03-23T20:53:00.037665+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T20:53:00.037665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f0363173-1820-4027-8318-a96cb3f0e5f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c6028d94-1be7-43bb-8c5c-999d150aa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8595a61f-9a3a-4503-9214-1f7dd7d385a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:00.285 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-03-23T20:53:00.285 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-23T20:53:00.487 INFO:tasks.ceph.cluster1.osd.2.smithi117.stderr:2024-03-23T20:53:00.486+0000 7f4ca0249640 -1 Falling back to public interface 2024-03-23T20:53:00.587 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-03-23T20:53:00.652 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:00.650+0000 7f2c0df5c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T20:53:00.690 INFO:tasks.ceph.cluster1.osd.1.smithi117.stderr:2024-03-23T20:53:00.690+0000 7f5d30310640 -1 Falling back to public interface 2024-03-23T20:53:00.698 INFO:tasks.ceph.cluster1.osd.0.smithi117.stderr:2024-03-23T20:53:00.698+0000 7f65b38e7640 -1 Falling back to public interface 2024-03-23T20:53:00.711 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:00.710+0000 7f2c0df5c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T20:53:00.764 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:00.762+0000 7f2c0df5c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T20:53:00.768 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-23T20:53:00.768 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":5,"fsid":"f60222f0-44b2-480a-bdc8-0eb22ebe7b33","created":"2024-03-23T20:52:59.149423+0000","modified":"2024-03-23T20:53:00.037665+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T20:53:00.037665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f0363173-1820-4027-8318-a96cb3f0e5f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c6028d94-1be7-43bb-8c5c-999d150aa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8595a61f-9a3a-4503-9214-1f7dd7d385a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:00.783 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-23T20:53:00.842 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:00.842+0000 7f2c0df5c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T20:53:00.894 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:00.894+0000 7f2c0df5c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T20:53:00.970 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:00.970+0000 7f2c0df5c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T20:53:01.020 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:01.018+0000 7f2c0df5c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T20:53:01.169 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:01.166+0000 7f2c0df5c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T20:53:01.228 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:01.226+0000 7f2c0df5c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T20:53:01.841 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-23T20:53:01.841 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-23T20:53:01.841 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr: from numpy import show_config as show_numpy_config 2024-03-23T20:53:01.912 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:01.910+0000 7f2c0df5c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T20:53:02.093 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:02.090+0000 7f2c0df5c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T20:53:02.257 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:02.254+0000 7f2c0df5c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T20:53:02.326 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:02.326+0000 7f2c0df5c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T20:53:02.470 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:02.470+0000 7f2c0df5c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T20:53:03.101 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:03.098+0000 7f2c0df5c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T20:53:03.175 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:03.174+0000 7f2c0df5c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T20:53:03.357 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:03.354+0000 7f2c0df5c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T20:53:03.430 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:03.430+0000 7f2c0df5c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T20:53:03.725 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:03.722+0000 7f2c0df5c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T20:53:03.790 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:03.790+0000 7f2c0df5c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T20:53:04.356 INFO:tasks.ceph.cluster1.osd.2.smithi117.stderr:2024-03-23T20:53:04.354+0000 7f4ca0249640 -1 osd.2 0 log_to_monitors true 2024-03-23T20:53:04.521 INFO:tasks.ceph.cluster1.osd.1.smithi117.stderr:2024-03-23T20:53:04.522+0000 7f5d30310640 -1 osd.1 0 log_to_monitors true 2024-03-23T20:53:04.635 INFO:tasks.ceph.cluster1.osd.0.smithi117.stderr:2024-03-23T20:53:04.634+0000 7f65b38e7640 -1 osd.0 0 log_to_monitors true 2024-03-23T20:53:05.028 INFO:tasks.ceph.cluster1.mgr.x.smithi117.stderr:2024-03-23T20:53:05.026+0000 7f2bb5612640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-23T20:53:07.086 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-03-23T20:53:07.474 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-23T20:53:07.474 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":7,"fsid":"f60222f0-44b2-480a-bdc8-0eb22ebe7b33","created":"2024-03-23T20:52:59.149423+0000","modified":"2024-03-23T20:53:06.100005+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T20:53:00.037665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f0363173-1820-4027-8318-a96cb3f0e5f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c6028d94-1be7-43bb-8c5c-999d150aa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8595a61f-9a3a-4503-9214-1f7dd7d385a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:07.496 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-23T20:53:08.072 INFO:tasks.ceph.cluster1.osd.0.smithi117.stderr:2024-03-23T20:53:08.070+0000 7f65af069640 -1 osd.0 0 waiting for initial osdmap 2024-03-23T20:53:08.077 INFO:tasks.ceph.cluster1.osd.2.smithi117.stderr:2024-03-23T20:53:08.074+0000 7f4c9b9cb640 -1 osd.2 0 waiting for initial osdmap 2024-03-23T20:53:08.098 INFO:tasks.ceph.cluster1.osd.0.smithi117.stderr:2024-03-23T20:53:08.098+0000 7f65a76ae640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T20:53:08.098 INFO:tasks.ceph.cluster1.osd.1.smithi117.stderr:2024-03-23T20:53:08.098+0000 7f5d2ba92640 -1 osd.1 0 waiting for initial osdmap 2024-03-23T20:53:08.114 INFO:tasks.ceph.cluster1.osd.2.smithi117.stderr:2024-03-23T20:53:08.114+0000 7f4c94010640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T20:53:08.138 INFO:tasks.ceph.cluster1.osd.1.smithi117.stderr:2024-03-23T20:53:08.138+0000 7f5d240d7640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T20:53:13.799 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-03-23T20:53:14.092 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-23T20:53:14.092 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"f60222f0-44b2-480a-bdc8-0eb22ebe7b33","created":"2024-03-23T20:52:59.149423+0000","modified":"2024-03-23T20:53:12.215377+0000","last_up_change":"2024-03-23T20:53:09.102717+0000","last_in_change":"2024-03-23T20:53:00.037665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T20:53:09.899996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0363173-1820-4027-8318-a96cb3f0e5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23953},{"type":"v1","addr":"172.21.15.117:6817","nonce":23953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23953},{"type":"v1","addr":"172.21.15.117:6819","nonce":23953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23953},{"type":"v1","addr":"172.21.15.117:6823","nonce":23953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23953},{"type":"v1","addr":"172.21.15.117:6821","nonce":23953}]},"public_addr":"172.21.15.117:6817/23953","cluster_addr":"172.21.15.117:6819/23953","heartbeat_back_addr":"172.21.15.117:6823/23953","heartbeat_front_addr":"172.21.15.117:6821/23953","state":["exists","up"]},{"osd":1,"uuid":"c6028d94-1be7-43bb-8c5c-999d150aa717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23952},{"type":"v1","addr":"172.21.15.117:6809","nonce":23952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23952},{"type":"v1","addr":"172.21.15.117:6811","nonce":23952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23952},{"type":"v1","addr":"172.21.15.117:6815","nonce":23952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23952},{"type":"v1","addr":"172.21.15.117:6813","nonce":23952}]},"public_addr":"172.21.15.117:6809/23952","cluster_addr":"172.21.15.117:6811/23952","heartbeat_back_addr":"172.21.15.117:6815/23952","heartbeat_front_addr":"172.21.15.117:6813/23952","state":["exists","up"]},{"osd":2,"uuid":"8595a61f-9a3a-4503-9214-1f7dd7d385a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":23954},{"type":"v1","addr":"172.21.15.117:6801","nonce":23954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23954},{"type":"v1","addr":"172.21.15.117:6803","nonce":23954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23954},{"type":"v1","addr":"172.21.15.117:6807","nonce":23954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23954},{"type":"v1","addr":"172.21.15.117:6805","nonce":23954}]},"public_addr":"172.21.15.117:6801/23954","cluster_addr":"172.21.15.117:6803/23954","heartbeat_back_addr":"172.21.15.117:6807/23954","heartbeat_front_addr":"172.21.15.117:6805/23954","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.628809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.575003+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.402693+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:14.109 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-23T20:53:14.109 INFO:tasks.ceph:Creating RBD pool 2024-03-23T20:53:14.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-03-23T20:53:15.233 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'rbd' created 2024-03-23T20:53:15.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-23T20:53:16.246 INFO:teuthology.orchestra.run.smithi117.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-23T20:53:16.276 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-03-23T20:53:16.277 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-03-23T20:53:16.277 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-03-23T20:53:16.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-03-23T20:53:16.278 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-23T20:53:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:16.627 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23776},{"type":"v1","addr":"172.21.15.117:6825","nonce":23776}]},"active_addr":"172.21.15.117:6825/23776","active_change":"2024-03-23T20:53:03.813965+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":2662959007}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":158828777}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":2876527170}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":2341825430}]}]} 2024-03-23T20:53:16.629 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-03-23T20:53:16.629 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-03-23T20:53:16.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-03-23T20:53:16.921 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:16.922 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":13,"fsid":"f60222f0-44b2-480a-bdc8-0eb22ebe7b33","created":"2024-03-23T20:52:59.149423+0000","modified":"2024-03-23T20:53:16.232555+0000","last_up_change":"2024-03-23T20:53:09.102717+0000","last_in_change":"2024-03-23T20:53:00.037665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T20:53:09.899996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-23T20:53:14.441478+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0363173-1820-4027-8318-a96cb3f0e5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23953},{"type":"v1","addr":"172.21.15.117:6817","nonce":23953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23953},{"type":"v1","addr":"172.21.15.117:6819","nonce":23953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23953},{"type":"v1","addr":"172.21.15.117:6823","nonce":23953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23953},{"type":"v1","addr":"172.21.15.117:6821","nonce":23953}]},"public_addr":"172.21.15.117:6817/23953","cluster_addr":"172.21.15.117:6819/23953","heartbeat_back_addr":"172.21.15.117:6823/23953","heartbeat_front_addr":"172.21.15.117:6821/23953","state":["exists","up"]},{"osd":1,"uuid":"c6028d94-1be7-43bb-8c5c-999d150aa717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23952},{"type":"v1","addr":"172.21.15.117:6809","nonce":23952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23952},{"type":"v1","addr":"172.21.15.117:6811","nonce":23952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23952},{"type":"v1","addr":"172.21.15.117:6815","nonce":23952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23952},{"type":"v1","addr":"172.21.15.117:6813","nonce":23952}]},"public_addr":"172.21.15.117:6809/23952","cluster_addr":"172.21.15.117:6811/23952","heartbeat_back_addr":"172.21.15.117:6815/23952","heartbeat_front_addr":"172.21.15.117:6813/23952","state":["exists","up"]},{"osd":2,"uuid":"8595a61f-9a3a-4503-9214-1f7dd7d385a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":23954},{"type":"v1","addr":"172.21.15.117:6801","nonce":23954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23954},{"type":"v1","addr":"172.21.15.117:6803","nonce":23954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23954},{"type":"v1","addr":"172.21.15.117:6807","nonce":23954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23954},{"type":"v1","addr":"172.21.15.117:6805","nonce":23954}]},"public_addr":"172.21.15.117:6801/23954","cluster_addr":"172.21.15.117:6803/23954","heartbeat_back_addr":"172.21.15.117:6807/23954","heartbeat_front_addr":"172.21.15.117:6805/23954","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.628809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.575003+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.402693+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:16.940 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-03-23T20:53:16.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-03-23T20:53:17.231 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:17.231 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":13,"fsid":"f60222f0-44b2-480a-bdc8-0eb22ebe7b33","created":"2024-03-23T20:52:59.149423+0000","modified":"2024-03-23T20:53:16.232555+0000","last_up_change":"2024-03-23T20:53:09.102717+0000","last_in_change":"2024-03-23T20:53:00.037665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T20:53:09.899996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-23T20:53:14.441478+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0363173-1820-4027-8318-a96cb3f0e5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23953},{"type":"v1","addr":"172.21.15.117:6817","nonce":23953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23953},{"type":"v1","addr":"172.21.15.117:6819","nonce":23953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23953},{"type":"v1","addr":"172.21.15.117:6823","nonce":23953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23953},{"type":"v1","addr":"172.21.15.117:6821","nonce":23953}]},"public_addr":"172.21.15.117:6817/23953","cluster_addr":"172.21.15.117:6819/23953","heartbeat_back_addr":"172.21.15.117:6823/23953","heartbeat_front_addr":"172.21.15.117:6821/23953","state":["exists","up"]},{"osd":1,"uuid":"c6028d94-1be7-43bb-8c5c-999d150aa717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23952},{"type":"v1","addr":"172.21.15.117:6809","nonce":23952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23952},{"type":"v1","addr":"172.21.15.117:6811","nonce":23952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23952},{"type":"v1","addr":"172.21.15.117:6815","nonce":23952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23952},{"type":"v1","addr":"172.21.15.117:6813","nonce":23952}]},"public_addr":"172.21.15.117:6809/23952","cluster_addr":"172.21.15.117:6811/23952","heartbeat_back_addr":"172.21.15.117:6815/23952","heartbeat_front_addr":"172.21.15.117:6813/23952","state":["exists","up"]},{"osd":2,"uuid":"8595a61f-9a3a-4503-9214-1f7dd7d385a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":23954},{"type":"v1","addr":"172.21.15.117:6801","nonce":23954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23954},{"type":"v1","addr":"172.21.15.117:6803","nonce":23954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23954},{"type":"v1","addr":"172.21.15.117:6807","nonce":23954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23954},{"type":"v1","addr":"172.21.15.117:6805","nonce":23954}]},"public_addr":"172.21.15.117:6801/23954","cluster_addr":"172.21.15.117:6803/23954","heartbeat_back_addr":"172.21.15.117:6807/23954","heartbeat_front_addr":"172.21.15.117:6805/23954","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.628809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.575003+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.402693+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:17.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-03-23T20:53:17.250 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-03-23T20:53:17.250 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-03-23T20:53:17.401 INFO:teuthology.orchestra.run.smithi117.stdout:34359738373 2024-03-23T20:53:17.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-03-23T20:53:17.402 INFO:teuthology.orchestra.run.smithi117.stdout:34359738374 2024-03-23T20:53:17.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-03-23T20:53:17.403 INFO:teuthology.orchestra.run.smithi117.stdout:34359738373 2024-03-23T20:53:17.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-03-23T20:53:17.719 INFO:teuthology.orchestra.run.smithi117.stdout:34359738372 2024-03-23T20:53:17.733 INFO:teuthology.orchestra.run.smithi117.stdout:34359738372 2024-03-23T20:53:17.736 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2024-03-23T20:53:17.740 INFO:teuthology.orchestra.run.smithi117.stdout:34359738373 2024-03-23T20:53:17.752 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-03-23T20:53:17.758 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.2 2024-03-23T20:53:18.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-03-23T20:53:18.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-03-23T20:53:18.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-03-23T20:53:19.050 INFO:teuthology.orchestra.run.smithi117.stdout:34359738373 2024-03-23T20:53:19.067 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2024-03-23T20:53:19.067 DEBUG:teuthology.parallel:result is None 2024-03-23T20:53:19.071 INFO:teuthology.orchestra.run.smithi117.stdout:34359738374 2024-03-23T20:53:19.079 INFO:teuthology.orchestra.run.smithi117.stdout:34359738373 2024-03-23T20:53:19.088 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2024-03-23T20:53:19.089 DEBUG:teuthology.parallel:result is None 2024-03-23T20:53:19.097 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2024-03-23T20:53:19.097 DEBUG:teuthology.parallel:result is None 2024-03-23T20:53:19.097 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-03-23T20:53:19.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-23T20:53:19.406 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:19.407 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-23T20:53:19.423 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-23T20:53:17.822051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82560,"kb_used_data":1492,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032960,"statfs":{"total":289910292480,"available":289825751040,"internally_reserved":0,"allocated":1527808,"data_stored":1273924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.589344"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.265365+0000","last_change":"2024-03-23T20:53:16.260778+0000","last_active":"2024-03-23T20:53:16.265365+0000","last_peered":"2024-03-23T20:53:16.265365+0000","last_clean":"2024-03-23T20:53:16.265365+0000","last_became_active":"2024-03-23T20:53:16.260429+0000","last_became_peered":"2024-03-23T20:53:16.260429+0000","last_unstale":"2024-03-23T20:53:16.265365+0000","last_undegraded":"2024-03-23T20:53:16.265365+0000","last_fullsized":"2024-03-23T20:53:16.265365+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:08:21.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.262211+0000","last_change":"2024-03-23T20:53:16.257714+0000","last_active":"2024-03-23T20:53:16.262211+0000","last_peered":"2024-03-23T20:53:16.262211+0000","last_clean":"2024-03-23T20:53:16.262211+0000","last_became_active":"2024-03-23T20:53:16.257287+0000","last_became_peered":"2024-03-23T20:53:16.257287+0000","last_unstale":"2024-03-23T20:53:16.262211+0000","last_undegraded":"2024-03-23T20:53:16.262211+0000","last_fullsized":"2024-03-23T20:53:16.262211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:25.116796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.265316+0000","last_change":"2024-03-23T20:53:16.260048+0000","last_active":"2024-03-23T20:53:16.265316+0000","last_peered":"2024-03-23T20:53:16.265316+0000","last_clean":"2024-03-23T20:53:16.265316+0000","last_became_active":"2024-03-23T20:53:16.259683+0000","last_became_peered":"2024-03-23T20:53:16.259683+0000","last_unstale":"2024-03-23T20:53:16.265316+0000","last_undegraded":"2024-03-23T20:53:16.265316+0000","last_fullsized":"2024-03-23T20:53:16.265316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:39:53.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.265358+0000","last_change":"2024-03-23T20:53:16.260803+0000","last_active":"2024-03-23T20:53:16.265358+0000","last_peered":"2024-03-23T20:53:16.265358+0000","last_clean":"2024-03-23T20:53:16.265358+0000","last_became_active":"2024-03-23T20:53:16.260458+0000","last_became_peered":"2024-03-23T20:53:16.260458+0000","last_unstale":"2024-03-23T20:53:16.265358+0000","last_undegraded":"2024-03-23T20:53:16.265358+0000","last_fullsized":"2024-03-23T20:53:16.265358+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:25:51.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.260153+0000","last_change":"2024-03-23T20:53:16.258624+0000","last_active":"2024-03-23T20:53:16.260153+0000","last_peered":"2024-03-23T20:53:16.260153+0000","last_clean":"2024-03-23T20:53:16.260153+0000","last_became_active":"2024-03-23T20:53:16.257825+0000","last_became_peered":"2024-03-23T20:53:16.257825+0000","last_unstale":"2024-03-23T20:53:16.260153+0000","last_undegraded":"2024-03-23T20:53:16.260153+0000","last_fullsized":"2024-03-23T20:53:16.260153+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:55.573156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:17.252309+0000","last_change":"2024-03-23T20:53:16.257583+0000","last_active":"2024-03-23T20:53:17.252309+0000","last_peered":"2024-03-23T20:53:17.252309+0000","last_clean":"2024-03-23T20:53:17.252309+0000","last_became_active":"2024-03-23T20:53:16.257206+0000","last_became_peered":"2024-03-23T20:53:16.257206+0000","last_unstale":"2024-03-23T20:53:17.252309+0000","last_undegraded":"2024-03-23T20:53:17.252309+0000","last_fullsized":"2024-03-23T20:53:17.252309+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:49:25.279880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:17.252344+0000","last_change":"2024-03-23T20:53:16.257895+0000","last_active":"2024-03-23T20:53:17.252344+0000","last_peered":"2024-03-23T20:53:17.252344+0000","last_clean":"2024-03-23T20:53:17.252344+0000","last_became_active":"2024-03-23T20:53:16.257435+0000","last_became_peered":"2024-03-23T20:53:16.257435+0000","last_unstale":"2024-03-23T20:53:17.252344+0000","last_undegraded":"2024-03-23T20:53:17.252344+0000","last_fullsized":"2024-03-23T20:53:17.252344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:20:33.429387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.265277+0000","last_change":"2024-03-23T20:53:16.259073+0000","last_active":"2024-03-23T20:53:16.265277+0000","last_peered":"2024-03-23T20:53:16.265277+0000","last_clean":"2024-03-23T20:53:16.265277+0000","last_became_active":"2024-03-23T20:53:16.258711+0000","last_became_peered":"2024-03-23T20:53:16.258711+0000","last_unstale":"2024-03-23T20:53:16.265277+0000","last_undegraded":"2024-03-23T20:53:16.265277+0000","last_fullsized":"2024-03-23T20:53:16.265277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:03:03.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-23T20:53:15.235416+0000","last_change":"2024-03-23T20:53:11.553674+0000","last_active":"2024-03-23T20:53:15.235416+0000","last_peered":"2024-03-23T20:53:15.235416+0000","last_clean":"2024-03-23T20:53:15.235416+0000","last_became_active":"2024-03-23T20:53:11.551476+0000","last_became_peered":"2024-03-23T20:53:11.551476+0000","last_unstale":"2024-03-23T20:53:15.235416+0000","last_undegraded":"2024-03-23T20:53:15.235416+0000","last_fullsized":"2024-03-23T20:53:15.235416+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_clean_scrub_stamp":"2024-03-23T20:53:10.128204+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:26.471264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":620918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T20:53:19.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-23T20:53:19.699 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:19.699 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-23T20:53:19.715 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-23T20:53:17.822051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82560,"kb_used_data":1492,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032960,"statfs":{"total":289910292480,"available":289825751040,"internally_reserved":0,"allocated":1527808,"data_stored":1273924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.589344"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.265365+0000","last_change":"2024-03-23T20:53:16.260778+0000","last_active":"2024-03-23T20:53:16.265365+0000","last_peered":"2024-03-23T20:53:16.265365+0000","last_clean":"2024-03-23T20:53:16.265365+0000","last_became_active":"2024-03-23T20:53:16.260429+0000","last_became_peered":"2024-03-23T20:53:16.260429+0000","last_unstale":"2024-03-23T20:53:16.265365+0000","last_undegraded":"2024-03-23T20:53:16.265365+0000","last_fullsized":"2024-03-23T20:53:16.265365+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:08:21.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.262211+0000","last_change":"2024-03-23T20:53:16.257714+0000","last_active":"2024-03-23T20:53:16.262211+0000","last_peered":"2024-03-23T20:53:16.262211+0000","last_clean":"2024-03-23T20:53:16.262211+0000","last_became_active":"2024-03-23T20:53:16.257287+0000","last_became_peered":"2024-03-23T20:53:16.257287+0000","last_unstale":"2024-03-23T20:53:16.262211+0000","last_undegraded":"2024-03-23T20:53:16.262211+0000","last_fullsized":"2024-03-23T20:53:16.262211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:25.116796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.265316+0000","last_change":"2024-03-23T20:53:16.260048+0000","last_active":"2024-03-23T20:53:16.265316+0000","last_peered":"2024-03-23T20:53:16.265316+0000","last_clean":"2024-03-23T20:53:16.265316+0000","last_became_active":"2024-03-23T20:53:16.259683+0000","last_became_peered":"2024-03-23T20:53:16.259683+0000","last_unstale":"2024-03-23T20:53:16.265316+0000","last_undegraded":"2024-03-23T20:53:16.265316+0000","last_fullsized":"2024-03-23T20:53:16.265316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:39:53.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.265358+0000","last_change":"2024-03-23T20:53:16.260803+0000","last_active":"2024-03-23T20:53:16.265358+0000","last_peered":"2024-03-23T20:53:16.265358+0000","last_clean":"2024-03-23T20:53:16.265358+0000","last_became_active":"2024-03-23T20:53:16.260458+0000","last_became_peered":"2024-03-23T20:53:16.260458+0000","last_unstale":"2024-03-23T20:53:16.265358+0000","last_undegraded":"2024-03-23T20:53:16.265358+0000","last_fullsized":"2024-03-23T20:53:16.265358+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:25:51.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.260153+0000","last_change":"2024-03-23T20:53:16.258624+0000","last_active":"2024-03-23T20:53:16.260153+0000","last_peered":"2024-03-23T20:53:16.260153+0000","last_clean":"2024-03-23T20:53:16.260153+0000","last_became_active":"2024-03-23T20:53:16.257825+0000","last_became_peered":"2024-03-23T20:53:16.257825+0000","last_unstale":"2024-03-23T20:53:16.260153+0000","last_undegraded":"2024-03-23T20:53:16.260153+0000","last_fullsized":"2024-03-23T20:53:16.260153+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:55.573156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:17.252309+0000","last_change":"2024-03-23T20:53:16.257583+0000","last_active":"2024-03-23T20:53:17.252309+0000","last_peered":"2024-03-23T20:53:17.252309+0000","last_clean":"2024-03-23T20:53:17.252309+0000","last_became_active":"2024-03-23T20:53:16.257206+0000","last_became_peered":"2024-03-23T20:53:16.257206+0000","last_unstale":"2024-03-23T20:53:17.252309+0000","last_undegraded":"2024-03-23T20:53:17.252309+0000","last_fullsized":"2024-03-23T20:53:17.252309+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:49:25.279880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:17.252344+0000","last_change":"2024-03-23T20:53:16.257895+0000","last_active":"2024-03-23T20:53:17.252344+0000","last_peered":"2024-03-23T20:53:17.252344+0000","last_clean":"2024-03-23T20:53:17.252344+0000","last_became_active":"2024-03-23T20:53:16.257435+0000","last_became_peered":"2024-03-23T20:53:16.257435+0000","last_unstale":"2024-03-23T20:53:17.252344+0000","last_undegraded":"2024-03-23T20:53:17.252344+0000","last_fullsized":"2024-03-23T20:53:17.252344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:20:33.429387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-23T20:53:16.265277+0000","last_change":"2024-03-23T20:53:16.259073+0000","last_active":"2024-03-23T20:53:16.265277+0000","last_peered":"2024-03-23T20:53:16.265277+0000","last_clean":"2024-03-23T20:53:16.265277+0000","last_became_active":"2024-03-23T20:53:16.258711+0000","last_became_peered":"2024-03-23T20:53:16.258711+0000","last_unstale":"2024-03-23T20:53:16.265277+0000","last_undegraded":"2024-03-23T20:53:16.265277+0000","last_fullsized":"2024-03-23T20:53:16.265277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:03:03.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-23T20:53:15.235416+0000","last_change":"2024-03-23T20:53:11.553674+0000","last_active":"2024-03-23T20:53:15.235416+0000","last_peered":"2024-03-23T20:53:15.235416+0000","last_clean":"2024-03-23T20:53:15.235416+0000","last_became_active":"2024-03-23T20:53:11.551476+0000","last_became_peered":"2024-03-23T20:53:11.551476+0000","last_unstale":"2024-03-23T20:53:15.235416+0000","last_undegraded":"2024-03-23T20:53:15.235416+0000","last_fullsized":"2024-03-23T20:53:15.235416+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_clean_scrub_stamp":"2024-03-23T20:53:10.128204+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:26.471264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":620918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T20:53:19.716 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-03-23T20:53:19.716 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-03-23T20:53:19.716 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-03-23T20:53:19.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-03-23T20:53:20.009 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:20.010 INFO:teuthology.orchestra.run.smithi117.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T20:53:20.027 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-03-23T20:53:20.027 INFO:teuthology.run_tasks:Running task ceph... 2024-03-23T20:53:20.036 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-03-23T20:53:20.037 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-23T20:53:20.037 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668', 'branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7617668', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri11-testing-2024-03-21-0851-reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 6304, 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'targets': {'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1jcoioqpma6131vrwi23FPocv1RZFh/qwKKJR0unECZSCSz46OXgHJIhVax58gSn+DqTttcf+hwf4JLjyGxfs=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP3lLbZ167RBEDXN5OmvB8wahaNtp28lVj7Ctgps/XZan0ZhVsbikIRpq37YPGsjrb4IZAdK/KxIOad73zDNz7Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_journal.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-22_13:19:01', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-23T20:53:20.038 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-03-23T20:53:20.040 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-03-23T20:53:20.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T20:53:20.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T20:53:20.102 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:20.102 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T20:53:20.150 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T20:53:20.150 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-03-23T20:53:20.198 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T20:53:20.198 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T20:53:20.198 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-03-23T20:53:20.198 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T20:53:20.198 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-23 20:47:35.539859511 +0000 2024-03-23T20:53:20.198 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-23 20:47:35.267861941 +0000 2024-03-23T20:53:20.199 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-23 20:47:35.267861941 +0000 2024-03-23T20:53:20.199 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-03-23T20:53:20.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T20:53:20.253 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-23T20:53:20.253 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-23T20:53:20.253 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000447966 s, 1.1 MB/s 2024-03-23T20:53:20.254 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T20:53:20.303 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-03-23T20:53:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T20:53:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T20:53:20.350 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-03-23T20:53:20.350 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T20:53:20.350 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-23 20:47:36.075854724 +0000 2024-03-23T20:53:20.350 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-23 20:47:35.779857368 +0000 2024-03-23T20:53:20.350 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-23 20:47:35.779857368 +0000 2024-03-23T20:53:20.350 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-03-23T20:53:20.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T20:53:20.403 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-23T20:53:20.404 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-23T20:53:20.404 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000323329 s, 1.6 MB/s 2024-03-23T20:53:20.405 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T20:53:20.455 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-03-23T20:53:20.502 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T20:53:20.502 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T20:53:20.502 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-03-23T20:53:20.502 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T20:53:20.502 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-23 20:47:36.587850150 +0000 2024-03-23T20:53:20.502 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-23 20:47:36.311852615 +0000 2024-03-23T20:53:20.502 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-23 20:47:36.311852615 +0000 2024-03-23T20:53:20.502 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-03-23T20:53:20.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T20:53:20.556 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-23T20:53:20.556 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-23T20:53:20.556 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000336046 s, 1.5 MB/s 2024-03-23T20:53:20.557 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T20:53:20.607 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-03-23T20:53:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T20:53:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T20:53:20.654 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-03-23T20:53:20.654 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T20:53:20.654 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-23 20:47:37.059845935 +0000 2024-03-23T20:53:20.655 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-23 20:47:36.811848150 +0000 2024-03-23T20:53:20.655 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-23 20:47:36.811848150 +0000 2024-03-23T20:53:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-03-23T20:53:20.655 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T20:53:20.709 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-23T20:53:20.709 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-23T20:53:20.709 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000382164 s, 1.3 MB/s 2024-03-23T20:53:20.710 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T20:53:20.759 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-23T20:53:20.759 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-23T20:53:20.759 INFO:tasks.ceph:Generating config... 2024-03-23T20:53:20.760 INFO:tasks.ceph:[client] debug rbd = 20 2024-03-23T20:53:20.760 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client] log to stderr = False 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client] rbd default features = 125 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:20.761 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:20.762 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:20.762 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:20.762 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:20.762 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-23T20:53:20.762 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-03-23T20:53:20.762 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-23T20:53:20.762 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-23T20:53:20.762 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-23T20:53:20.762 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-23T20:53:20.762 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-23T20:53:20.762 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-23T20:53:20.762 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-23T20:53:20.762 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-23T20:53:20.763 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-23T20:53:20.764 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-03-23T20:53:20.764 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-03-23T20:53:20.833 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.keyring 2024-03-23T20:53:20.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-03-23T20:53:20.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-03-23T20:53:20.965 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.204')] 2024-03-23T20:53:20.965 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.204', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-03-23T20:53:20.966 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:20.966 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T20:53:21.014 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.204 --print /home/ubuntu/cephtest/cluster2.monmap 2024-03-23T20:53:21.093 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-03-23T20:53:21.093 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: generated fsid d781caf7-4325-48b0-8e95-908c0fb74304 2024-03-23T20:53:21.093 INFO:teuthology.orchestra.run.smithi204.stdout:setting min_mon_release = pacific 2024-03-23T20:53:21.093 INFO:teuthology.orchestra.run.smithi204.stdout:epoch 0 2024-03-23T20:53:21.093 INFO:teuthology.orchestra.run.smithi204.stdout:fsid d781caf7-4325-48b0-8e95-908c0fb74304 2024-03-23T20:53:21.093 INFO:teuthology.orchestra.run.smithi204.stdout:last_changed 2024-03-23T20:53:21.094439+0000 2024-03-23T20:53:21.093 INFO:teuthology.orchestra.run.smithi204.stdout:created 2024-03-23T20:53:21.094439+0000 2024-03-23T20:53:21.094 INFO:teuthology.orchestra.run.smithi204.stdout:min_mon_release 16 (pacific) 2024-03-23T20:53:21.094 INFO:teuthology.orchestra.run.smithi204.stdout:election_strategy: 1 2024-03-23T20:53:21.094 INFO:teuthology.orchestra.run.smithi204.stdout:0: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.a 2024-03-23T20:53:21.094 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-03-23T20:53:21.095 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T20:53:21.142 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID d781caf7-4325-48b0-8e95-908c0fb74304... 2024-03-23T20:53:21.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-03-23T20:53:21.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-03-23T20:53:21.188 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-03-23T20:53:21.189 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-03-23T20:53:21.190 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = 172.21.15.204 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: mon client directed command retry = 5 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on pool no app = False 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: ms inject socket failures = 5000 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = d781caf7-4325-48b0-8e95-908c0fb74304 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-03-23T20:53:21.191 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.192 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: bdev async discard = True 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: bdev enable discard = True 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore allocator = stupid 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 1/20 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 1/20 2024-03-23T20:53:21.193 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 4/10 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-23T20:53:21.194 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.195 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-03-23T20:53:21.196 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: debug rbd = 20 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: debug rbd_mirror = 15 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: log to stderr = False 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: rbd default features = 125 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.0] 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.1] 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.197 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.2] 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.3] 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.4] 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.5] 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout:[client.mirror.6] 2024-03-23T20:53:21.198 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.199 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.215 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-03-23T20:53:21.215 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-03-23T20:53:21.216 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-03-23T20:53:21.217 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-03-23T20:53:21.218 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = 172.21.15.204 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on pool no app = False 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = d781caf7-4325-48b0-8e95-908c0fb74304 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.219 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.220 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: bdev async discard = True 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: bdev enable discard = True 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore allocator = stupid 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 1/20 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 1/20 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-23T20:53:21.221 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 4/10 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-03-23T20:53:21.222 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-03-23T20:53:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.224 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd = 20 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd_mirror = 15 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: log to stderr = False 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: rbd default features = 125 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.225 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.4] 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.5] 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.226 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.227 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.6] 2024-03-23T20:53:21.227 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-03-23T20:53:21.227 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-03-23T20:53:21.227 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-03-23T20:53:21.228 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-03-23T20:53:21.315 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-23T20:53:21.315 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:21.316 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-03-23T20:53:21.321 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:21.321 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-03-23T20:53:21.370 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T20:53:21.370 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:53:21.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/cluster2.keyring 2024-03-23T20:53:21.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-03-23T20:53:21.393 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:53:21.393 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-03-23T20:53:21.442 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T20:53:21.442 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:21.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/cluster2.keyring 2024-03-23T20:53:21.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-03-23T20:53:21.465 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:21.465 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-03-23T20:53:21.514 INFO:tasks.ceph:Setting up mon nodes... 2024-03-23T20:53:21.514 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-23T20:53:21.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-03-23T20:53:21.556 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-03-23T20:53:21.559 INFO:tasks.ceph:Setting up mds nodes... 2024-03-23T20:53:21.559 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-23T20:53:21.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-03-23T20:53:21.635 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-03-23T20:53:21.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-03-23T20:53:21.682 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-03-23T20:53:21.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-03-23T20:53:21.768 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-03-23T20:53:21.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-03-23T20:53:21.851 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-03-23T20:53:21.861 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-03-23T20:53:21.938 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-03-23T20:53:21.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-03-23T20:53:22.024 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-03-23T20:53:22.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-03-23T20:53:22.114 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-03-23T20:53:22.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-03-23T20:53:22.199 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-03-23T20:53:22.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-03-23T20:53:22.286 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-03-23T20:53:22.296 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-23T20:53:22.297 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-23T20:53:22.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-03-23T20:53:22.349 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-23T20:53:22.349 INFO:tasks.ceph:role: cluster2.osd.0 2024-03-23T20:53:22.349 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T20:53:22.350 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-23T20:53:22.414 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T20:53:22.415 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T20:53:22.898 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-23T20:53:22.900 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-23T20:53:22.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-03-23T20:53:23.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-03-23T20:53:23.061 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-03-23T20:53:23.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:53:23.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-03-23T20:53:23.117 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-23T20:53:23.117 INFO:tasks.ceph:role: cluster2.osd.1 2024-03-23T20:53:23.118 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T20:53:23.118 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T20:53:23.180 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T20:53:23.181 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T20:53:23.618 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-23T20:53:23.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-23T20:53:23.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-03-23T20:53:23.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-03-23T20:53:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:53:23.720 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-03-23T20:53:23.721 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-03-23T20:53:23.771 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-03-23T20:53:23.771 INFO:tasks.ceph:role: cluster2.osd.2 2024-03-23T20:53:23.771 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T20:53:23.772 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T20:53:23.832 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T20:53:23.833 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T20:53:24.294 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-23T20:53:24.296 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-23T20:53:24.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-03-23T20:53:24.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-03-23T20:53:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:53:24.430 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-03-23T20:53:24.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-03-23T20:53:24.678 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:24.678+0000 7fa28f0de640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-03-23T20:53:24.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:24.686+0000 7fa28f0de640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-03-23T20:53:24.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:24.686+0000 7fa28f0de640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-03-23T20:53:24.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:24.686+0000 7fa28f0de640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-03-23T20:53:24.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:24.686+0000 7fa28f0de640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-03-23T20:53:24.693 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:24.694+0000 7fa28f0de640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-03-23T20:53:27.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-03-23T20:53:27.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-03-23T20:53:27.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:27.778+0000 7f73c77cf640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-03-23T20:53:27.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:27.778+0000 7f73c77cf640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-03-23T20:53:27.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:27.778+0000 7f73c77cf640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-03-23T20:53:27.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:27.778+0000 7f73c77cf640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-03-23T20:53:27.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:27.778+0000 7f73c77cf640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-03-23T20:53:27.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:27.778+0000 7f73c77cf640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-03-23T20:53:30.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-03-23T20:53:30.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-03-23T20:53:30.887 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:30.886+0000 7f7689db0640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-03-23T20:53:30.887 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:30.886+0000 7f7689db0640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-03-23T20:53:30.888 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:30.886+0000 7f7689db0640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-03-23T20:53:30.888 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:30.886+0000 7f7689db0640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-03-23T20:53:30.888 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:30.886+0000 7f7689db0640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-03-23T20:53:30.888 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-23T20:53:30.886+0000 7f7689db0640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-03-23T20:53:33.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-03-23T20:53:33.910 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-23T20:53:33.910 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:53:33.911 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-03-23T20:53:33.926 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:33.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-03-23T20:53:33.965 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:33.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-03-23T20:53:34.026 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-03-23T20:53:34.082 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-23T20:53:34.082 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-03-23T20:53:34.088 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.088 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-03-23T20:53:34.130 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.130 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-03-23T20:53:34.178 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.178 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-03-23T20:53:34.226 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.226 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-03-23T20:53:34.274 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.275 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-03-23T20:53:34.322 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.322 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-03-23T20:53:34.370 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.371 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-03-23T20:53:34.418 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:34.419 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-03-23T20:53:34.467 INFO:tasks.ceph:Adding keys to all mons... 2024-03-23T20:53:34.467 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/cluster2.keyring 2024-03-23T20:53:34.520 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBBQf9lRVBfIRAAwbx0UdzWkCxQgtF4YkPwYg== 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBEQf9lMfCEKBAADN/A5yl/ppSmD4hNSJdiNw== 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBHQf9llPR9LhAABIn2MEPXBs3jVlVtKCwUDA== 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBKQf9liezyNBAAl5UEwfkryp+q55pWuUwg+w== 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBBQf9ltaoLJhAAeGtuLbzXmYsxhQvFMXom9Q== 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror] 2024-03-23T20:53:34.521 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBBQf9lING+KBAA3M2T5mqjTx7hLqNyJ22zyA== 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBBQf9l0rDXLRAAB+nOYgyFF9koYIp9OkPn5Q== 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBBQf9ldFHUMhAADmAwWvFPd5Uz0xADy9+NhQ== 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBBQf9lueUFOBAA+xGd+rWRdOkuVLdfcMOOPg== 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBCQf9l7oaNARAAWr8qhtsXggtPEb/xLZ4X7g== 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.4] 2024-03-23T20:53:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBCQf9lD73sBhAAFtY+pm46RZbMvn8EWLkdfQ== 2024-03-23T20:53:34.523 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.5] 2024-03-23T20:53:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBCQf9lM5n1CxAAINPAQxSuzzq0zxiDUMdEaQ== 2024-03-23T20:53:34.523 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.6] 2024-03-23T20:53:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBCQf9lYJgjERAAmvwgHxv4WconRVAx0OCiPg== 2024-03-23T20:53:34.524 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T20:53:34.609 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T20:53:34.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T20:53:34.785 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T20:53:34.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:34.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:35.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:35.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:35.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:35.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:35.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:35.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:35.486 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T20:53:35.530 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-23T20:53:35.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-03-23T20:53:35.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-03-23T20:53:35.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-03-23T20:53:35.724 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-03-23T20:53:35.769 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-03-23T20:53:35.769 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-03-23T20:53:35.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-03-23T20:53:35.815 INFO:tasks.ceph.cluster2.mon.a:Started 2024-03-23T20:53:35.815 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-03-23T20:53:35.815 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-03-23T20:53:35.815 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-03-23T20:53:35.818 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-03-23T20:53:35.818 INFO:tasks.ceph:Setting crush tunables to default 2024-03-23T20:53:35.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd crush tunables default 2024-03-23T20:53:36.065 INFO:teuthology.orchestra.run.smithi204.stderr:adjusted tunables profile to default 2024-03-23T20:53:36.083 INFO:tasks.ceph:check_enable_crimson: False 2024-03-23T20:53:36.083 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-03-23T20:53:36.083 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:36.084 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-03-23T20:53:36.095 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:36.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-03-23T20:53:36.148 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:53:36.148 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-03-23T20:53:36.200 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new 06c6a838-cd4d-4e46-8a3a-97ec64073385 0 2024-03-23T20:53:36.213 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:36.214+0000 7f5ba24d6180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T20:53:36.463 INFO:teuthology.orchestra.run.smithi204.stdout:0 2024-03-23T20:53:36.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new 25daa7ad-a344-420f-93ab-040a61bf446c 1 2024-03-23T20:53:36.607 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:36.606+0000 7f5ba24d6180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T20:53:36.686 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-23T20:53:36.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new caa8e5a6-7291-4e08-9335-354bbd6e892f 2 2024-03-23T20:53:36.734 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:36.734+0000 7f5ba24d6180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T20:53:36.908 INFO:teuthology.orchestra.run.smithi204.stdout:2 2024-03-23T20:53:36.927 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-03-23T20:53:36.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-03-23T20:53:36.929 INFO:tasks.ceph.cluster2.osd.0:Started 2024-03-23T20:53:37.015 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-03-23T20:53:37.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-03-23T20:53:37.019 INFO:tasks.ceph.cluster2.osd.1:Started 2024-03-23T20:53:37.019 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-03-23T20:53:37.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-03-23T20:53:37.022 INFO:tasks.ceph.cluster2.osd.2:Started 2024-03-23T20:53:37.022 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-03-23T20:53:37.138 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.138+0000 7f5ba24d6180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T20:53:37.187 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.186+0000 7f5ba24d6180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T20:53:37.204 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:37.204 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":5,"fsid":"d781caf7-4325-48b0-8e95-908c0fb74304","created":"2024-03-23T20:53:35.958816+0000","modified":"2024-03-23T20:53:36.908224+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T20:53:36.908224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"06c6a838-cd4d-4e46-8a3a-97ec64073385","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"25daa7ad-a344-420f-93ab-040a61bf446c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"caa8e5a6-7291-4e08-9335-354bbd6e892f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:37.220 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-03-23T20:53:37.220 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-23T20:53:37.234 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.234+0000 7f5ba24d6180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T20:53:37.307 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.306+0000 7f5ba24d6180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T20:53:37.355 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.354+0000 7f5ba24d6180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T20:53:37.426 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.426+0000 7f5ba24d6180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T20:53:37.474 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.474+0000 7f5ba24d6180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T20:53:37.523 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-03-23T20:53:37.524 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-03-23T20:53:37.522+0000 7f50aa86e640 -1 Falling back to public interface 2024-03-23T20:53:37.609 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.606+0000 7f5ba24d6180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T20:53:37.638 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-03-23T20:53:37.638+0000 7f7f86403640 -1 Falling back to public interface 2024-03-23T20:53:37.654 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-03-23T20:53:37.654+0000 7fe6907e1640 -1 Falling back to public interface 2024-03-23T20:53:37.658 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.658+0000 7f5ba24d6180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T20:53:37.700 INFO:teuthology.misc.health.smithi204.stdout: 2024-03-23T20:53:37.700 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":5,"fsid":"d781caf7-4325-48b0-8e95-908c0fb74304","created":"2024-03-23T20:53:35.958816+0000","modified":"2024-03-23T20:53:36.908224+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T20:53:36.908224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"06c6a838-cd4d-4e46-8a3a-97ec64073385","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"25daa7ad-a344-420f-93ab-040a61bf446c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"caa8e5a6-7291-4e08-9335-354bbd6e892f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:37.714 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-23T20:53:37.873 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-23T20:53:37.874 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-23T20:53:37.874 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr: from numpy import show_config as show_numpy_config 2024-03-23T20:53:37.881 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:37.882+0000 7f5ba24d6180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T20:53:38.010 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:38.010+0000 7f5ba24d6180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T20:53:38.116 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:38.114+0000 7f5ba24d6180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T20:53:38.176 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:38.174+0000 7f5ba24d6180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T20:53:38.305 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:38.306+0000 7f5ba24d6180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T20:53:38.768 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:38.766+0000 7f5ba24d6180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T20:53:38.840 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:38.838+0000 7f5ba24d6180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T20:53:38.980 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:38.978+0000 7f5ba24d6180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T20:53:39.051 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:39.050+0000 7f5ba24d6180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T20:53:39.326 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:39.326+0000 7f5ba24d6180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T20:53:39.379 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:39.378+0000 7f5ba24d6180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T20:53:40.409 INFO:tasks.ceph.cluster2.mgr.x.smithi117.stderr:2024-03-23T20:53:40.410+0000 7f5b49a91640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-23T20:53:41.220 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-03-23T20:53:41.218+0000 7f50aa86e640 -1 osd.0 0 log_to_monitors true 2024-03-23T20:53:41.399 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-03-23T20:53:41.398+0000 7fe6907e1640 -1 osd.1 0 log_to_monitors true 2024-03-23T20:53:41.413 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-03-23T20:53:41.410+0000 7f7f86403640 -1 osd.2 0 log_to_monitors true 2024-03-23T20:53:44.016 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-03-23T20:53:44.444 INFO:teuthology.misc.health.smithi204.stdout: 2024-03-23T20:53:44.445 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":7,"fsid":"d781caf7-4325-48b0-8e95-908c0fb74304","created":"2024-03-23T20:53:35.958816+0000","modified":"2024-03-23T20:53:42.901440+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T20:53:36.908224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"06c6a838-cd4d-4e46-8a3a-97ec64073385","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"25daa7ad-a344-420f-93ab-040a61bf446c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"caa8e5a6-7291-4e08-9335-354bbd6e892f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:44.470 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-23T20:53:44.884 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-03-23T20:53:44.882+0000 7f50a782b640 -1 osd.0 0 waiting for initial osdmap 2024-03-23T20:53:44.919 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-03-23T20:53:44.918+0000 7f509e635640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T20:53:44.933 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-03-23T20:53:44.930+0000 7fe68d79e640 -1 osd.1 0 waiting for initial osdmap 2024-03-23T20:53:44.945 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-03-23T20:53:44.946+0000 7f7f81b85640 -1 osd.2 0 waiting for initial osdmap 2024-03-23T20:53:44.962 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-03-23T20:53:44.962+0000 7fe6845a8640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T20:53:44.970 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-03-23T20:53:44.970+0000 7f7f7a1ca640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T20:53:50.773 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-03-23T20:53:51.071 INFO:teuthology.misc.health.smithi204.stdout: 2024-03-23T20:53:51.071 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":12,"fsid":"d781caf7-4325-48b0-8e95-908c0fb74304","created":"2024-03-23T20:53:35.958816+0000","modified":"2024-03-23T20:53:49.961855+0000","last_up_change":"2024-03-23T20:53:45.935300+0000","last_in_change":"2024-03-23T20:53:36.908224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T20:53:47.483126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06c6a838-cd4d-4e46-8a3a-97ec64073385","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":24013},{"type":"v1","addr":"172.21.15.204:6801","nonce":24013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":24013},{"type":"v1","addr":"172.21.15.204:6803","nonce":24013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":24013},{"type":"v1","addr":"172.21.15.204:6807","nonce":24013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":24013},{"type":"v1","addr":"172.21.15.204:6805","nonce":24013}]},"public_addr":"172.21.15.204:6801/24013","cluster_addr":"172.21.15.204:6803/24013","heartbeat_back_addr":"172.21.15.204:6807/24013","heartbeat_front_addr":"172.21.15.204:6805/24013","state":["exists","up"]},{"osd":1,"uuid":"25daa7ad-a344-420f-93ab-040a61bf446c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":24039},{"type":"v1","addr":"172.21.15.204:6817","nonce":24039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":24039},{"type":"v1","addr":"172.21.15.204:6819","nonce":24039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":24039},{"type":"v1","addr":"172.21.15.204:6823","nonce":24039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":24039},{"type":"v1","addr":"172.21.15.204:6821","nonce":24039}]},"public_addr":"172.21.15.204:6817/24039","cluster_addr":"172.21.15.204:6819/24039","heartbeat_back_addr":"172.21.15.204:6823/24039","heartbeat_front_addr":"172.21.15.204:6821/24039","state":["exists","up"]},{"osd":2,"uuid":"caa8e5a6-7291-4e08-9335-354bbd6e892f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":24038},{"type":"v1","addr":"172.21.15.204:6809","nonce":24038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":24038},{"type":"v1","addr":"172.21.15.204:6811","nonce":24038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":24038},{"type":"v1","addr":"172.21.15.204:6815","nonce":24038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":24038},{"type":"v1","addr":"172.21.15.204:6813","nonce":24038}]},"public_addr":"172.21.15.204:6809/24038","cluster_addr":"172.21.15.204:6811/24038","heartbeat_back_addr":"172.21.15.204:6815/24038","heartbeat_front_addr":"172.21.15.204:6813/24038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.182167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.449795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.391287+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:51.089 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-23T20:53:51.089 INFO:tasks.ceph:Creating RBD pool 2024-03-23T20:53:51.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-03-23T20:53:51.985 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'rbd' created 2024-03-23T20:53:52.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-23T20:53:52.986 INFO:teuthology.orchestra.run.smithi204.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-23T20:53:53.012 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-03-23T20:53:53.012 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-03-23T20:53:53.012 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-03-23T20:53:53.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-03-23T20:53:53.013 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-23T20:53:53.324 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:53.341 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":29041},{"type":"v1","addr":"172.21.15.117:6827","nonce":29041}]},"active_addr":"172.21.15.117:6827/29041","active_change":"2024-03-23T20:53:39.400168+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":1101489792}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":1394642696}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":3684255339}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":3001874426}]}]} 2024-03-23T20:53:53.344 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-03-23T20:53:53.344 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-03-23T20:53:53.344 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-03-23T20:53:53.622 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:53.622 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":14,"fsid":"d781caf7-4325-48b0-8e95-908c0fb74304","created":"2024-03-23T20:53:35.958816+0000","modified":"2024-03-23T20:53:52.980288+0000","last_up_change":"2024-03-23T20:53:45.935300+0000","last_in_change":"2024-03-23T20:53:36.908224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T20:53:47.483126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-23T20:53:51.400689+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06c6a838-cd4d-4e46-8a3a-97ec64073385","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":24013},{"type":"v1","addr":"172.21.15.204:6801","nonce":24013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":24013},{"type":"v1","addr":"172.21.15.204:6803","nonce":24013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":24013},{"type":"v1","addr":"172.21.15.204:6807","nonce":24013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":24013},{"type":"v1","addr":"172.21.15.204:6805","nonce":24013}]},"public_addr":"172.21.15.204:6801/24013","cluster_addr":"172.21.15.204:6803/24013","heartbeat_back_addr":"172.21.15.204:6807/24013","heartbeat_front_addr":"172.21.15.204:6805/24013","state":["exists","up"]},{"osd":1,"uuid":"25daa7ad-a344-420f-93ab-040a61bf446c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":24039},{"type":"v1","addr":"172.21.15.204:6817","nonce":24039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":24039},{"type":"v1","addr":"172.21.15.204:6819","nonce":24039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":24039},{"type":"v1","addr":"172.21.15.204:6823","nonce":24039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":24039},{"type":"v1","addr":"172.21.15.204:6821","nonce":24039}]},"public_addr":"172.21.15.204:6817/24039","cluster_addr":"172.21.15.204:6819/24039","heartbeat_back_addr":"172.21.15.204:6823/24039","heartbeat_front_addr":"172.21.15.204:6821/24039","state":["exists","up"]},{"osd":2,"uuid":"caa8e5a6-7291-4e08-9335-354bbd6e892f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":24038},{"type":"v1","addr":"172.21.15.204:6809","nonce":24038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":24038},{"type":"v1","addr":"172.21.15.204:6811","nonce":24038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":24038},{"type":"v1","addr":"172.21.15.204:6815","nonce":24038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":24038},{"type":"v1","addr":"172.21.15.204:6813","nonce":24038}]},"public_addr":"172.21.15.204:6809/24038","cluster_addr":"172.21.15.204:6811/24038","heartbeat_back_addr":"172.21.15.204:6815/24038","heartbeat_front_addr":"172.21.15.204:6813/24038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.182167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.449795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.391287+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:53.640 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-03-23T20:53:53.640 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-03-23T20:53:53.914 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:53.914 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":14,"fsid":"d781caf7-4325-48b0-8e95-908c0fb74304","created":"2024-03-23T20:53:35.958816+0000","modified":"2024-03-23T20:53:52.980288+0000","last_up_change":"2024-03-23T20:53:45.935300+0000","last_in_change":"2024-03-23T20:53:36.908224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T20:53:47.483126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-23T20:53:51.400689+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06c6a838-cd4d-4e46-8a3a-97ec64073385","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":24013},{"type":"v1","addr":"172.21.15.204:6801","nonce":24013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":24013},{"type":"v1","addr":"172.21.15.204:6803","nonce":24013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":24013},{"type":"v1","addr":"172.21.15.204:6807","nonce":24013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":24013},{"type":"v1","addr":"172.21.15.204:6805","nonce":24013}]},"public_addr":"172.21.15.204:6801/24013","cluster_addr":"172.21.15.204:6803/24013","heartbeat_back_addr":"172.21.15.204:6807/24013","heartbeat_front_addr":"172.21.15.204:6805/24013","state":["exists","up"]},{"osd":1,"uuid":"25daa7ad-a344-420f-93ab-040a61bf446c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":24039},{"type":"v1","addr":"172.21.15.204:6817","nonce":24039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":24039},{"type":"v1","addr":"172.21.15.204:6819","nonce":24039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":24039},{"type":"v1","addr":"172.21.15.204:6823","nonce":24039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":24039},{"type":"v1","addr":"172.21.15.204:6821","nonce":24039}]},"public_addr":"172.21.15.204:6817/24039","cluster_addr":"172.21.15.204:6819/24039","heartbeat_back_addr":"172.21.15.204:6823/24039","heartbeat_front_addr":"172.21.15.204:6821/24039","state":["exists","up"]},{"osd":2,"uuid":"caa8e5a6-7291-4e08-9335-354bbd6e892f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":24038},{"type":"v1","addr":"172.21.15.204:6809","nonce":24038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":24038},{"type":"v1","addr":"172.21.15.204:6811","nonce":24038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":24038},{"type":"v1","addr":"172.21.15.204:6815","nonce":24038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":24038},{"type":"v1","addr":"172.21.15.204:6813","nonce":24038}]},"public_addr":"172.21.15.204:6809/24038","cluster_addr":"172.21.15.204:6811/24038","heartbeat_back_addr":"172.21.15.204:6815/24038","heartbeat_front_addr":"172.21.15.204:6813/24038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.182167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.449795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.391287+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T20:53:53.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-03-23T20:53:53.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-03-23T20:53:53.933 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-03-23T20:53:54.086 INFO:teuthology.orchestra.run.smithi204.stdout:34359738375 2024-03-23T20:53:54.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-03-23T20:53:54.089 INFO:teuthology.orchestra.run.smithi204.stdout:34359738374 2024-03-23T20:53:54.089 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-03-23T20:53:54.090 INFO:teuthology.orchestra.run.smithi204.stdout:34359738373 2024-03-23T20:53:54.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-03-23T20:53:54.389 INFO:teuthology.orchestra.run.smithi204.stdout:34359738374 2024-03-23T20:53:54.406 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738374 for osd.0 2024-03-23T20:53:54.416 INFO:teuthology.orchestra.run.smithi204.stdout:34359738373 2024-03-23T20:53:54.416 INFO:teuthology.orchestra.run.smithi204.stdout:34359738372 2024-03-23T20:53:54.436 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-03-23T20:53:54.436 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-03-23T20:53:55.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-03-23T20:53:55.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-03-23T20:53:55.437 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-03-23T20:53:55.728 INFO:teuthology.orchestra.run.smithi204.stdout:34359738374 2024-03-23T20:53:55.734 INFO:teuthology.orchestra.run.smithi204.stdout:34359738373 2024-03-23T20:53:55.747 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738374 for osd.0 2024-03-23T20:53:55.752 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-03-23T20:53:55.761 INFO:teuthology.orchestra.run.smithi204.stdout:34359738372 2024-03-23T20:53:55.778 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-03-23T20:53:56.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-03-23T20:53:56.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-03-23T20:53:56.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-03-23T20:53:57.049 INFO:teuthology.orchestra.run.smithi204.stdout:34359738375 2024-03-23T20:53:57.066 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738375 for osd.0 2024-03-23T20:53:57.066 DEBUG:teuthology.parallel:result is None 2024-03-23T20:53:57.074 INFO:teuthology.orchestra.run.smithi204.stdout:34359738373 2024-03-23T20:53:57.092 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2024-03-23T20:53:57.092 INFO:teuthology.orchestra.run.smithi204.stdout:34359738374 2024-03-23T20:53:57.092 DEBUG:teuthology.parallel:result is None 2024-03-23T20:53:57.108 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-03-23T20:53:57.108 DEBUG:teuthology.parallel:result is None 2024-03-23T20:53:57.108 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-03-23T20:53:57.108 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-23T20:53:57.416 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:57.416 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-03-23T20:53:57.432 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-23T20:53:57.412778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82604,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032916,"statfs":{"total":289910292480,"available":289825705984,"internally_reserved":0,"allocated":1581056,"data_stored":1292330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.427752"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069234+0000","last_change":"2024-03-23T20:53:52.996000+0000","last_active":"2024-03-23T20:53:54.069234+0000","last_peered":"2024-03-23T20:53:54.069234+0000","last_clean":"2024-03-23T20:53:54.069234+0000","last_became_active":"2024-03-23T20:53:52.995659+0000","last_became_peered":"2024-03-23T20:53:52.995659+0000","last_unstale":"2024-03-23T20:53:54.069234+0000","last_undegraded":"2024-03-23T20:53:54.069234+0000","last_fullsized":"2024-03-23T20:53:54.069234+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:26.628717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069218+0000","last_change":"2024-03-23T20:53:52.995652+0000","last_active":"2024-03-23T20:53:54.069218+0000","last_peered":"2024-03-23T20:53:54.069218+0000","last_clean":"2024-03-23T20:53:54.069218+0000","last_became_active":"2024-03-23T20:53:52.995094+0000","last_became_peered":"2024-03-23T20:53:52.995094+0000","last_unstale":"2024-03-23T20:53:54.069218+0000","last_undegraded":"2024-03-23T20:53:54.069218+0000","last_fullsized":"2024-03-23T20:53:54.069218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:18.937950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069198+0000","last_change":"2024-03-23T20:53:52.994376+0000","last_active":"2024-03-23T20:53:54.069198+0000","last_peered":"2024-03-23T20:53:54.069198+0000","last_clean":"2024-03-23T20:53:54.069198+0000","last_became_active":"2024-03-23T20:53:52.994087+0000","last_became_peered":"2024-03-23T20:53:52.994087+0000","last_unstale":"2024-03-23T20:53:54.069198+0000","last_undegraded":"2024-03-23T20:53:54.069198+0000","last_fullsized":"2024-03-23T20:53:54.069198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:27.484370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069181+0000","last_change":"2024-03-23T20:53:52.995843+0000","last_active":"2024-03-23T20:53:54.069181+0000","last_peered":"2024-03-23T20:53:54.069181+0000","last_clean":"2024-03-23T20:53:54.069181+0000","last_became_active":"2024-03-23T20:53:52.995018+0000","last_became_peered":"2024-03-23T20:53:52.995018+0000","last_unstale":"2024-03-23T20:53:54.069181+0000","last_undegraded":"2024-03-23T20:53:54.069181+0000","last_fullsized":"2024-03-23T20:53:54.069181+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:02.274898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-23T20:53:53.994334+0000","last_change":"2024-03-23T20:53:52.993885+0000","last_active":"2024-03-23T20:53:53.994334+0000","last_peered":"2024-03-23T20:53:53.994334+0000","last_clean":"2024-03-23T20:53:53.994334+0000","last_became_active":"2024-03-23T20:53:52.993597+0000","last_became_peered":"2024-03-23T20:53:52.993597+0000","last_unstale":"2024-03-23T20:53:53.994334+0000","last_undegraded":"2024-03-23T20:53:53.994334+0000","last_fullsized":"2024-03-23T20:53:53.994334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:16.901421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.070620+0000","last_change":"2024-03-23T20:53:52.994757+0000","last_active":"2024-03-23T20:53:54.070620+0000","last_peered":"2024-03-23T20:53:54.070620+0000","last_clean":"2024-03-23T20:53:54.070620+0000","last_became_active":"2024-03-23T20:53:52.994351+0000","last_became_peered":"2024-03-23T20:53:52.994351+0000","last_unstale":"2024-03-23T20:53:54.070620+0000","last_undegraded":"2024-03-23T20:53:54.070620+0000","last_fullsized":"2024-03-23T20:53:54.070620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:46.671091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.070605+0000","last_change":"2024-03-23T20:53:52.993925+0000","last_active":"2024-03-23T20:53:54.070605+0000","last_peered":"2024-03-23T20:53:54.070605+0000","last_clean":"2024-03-23T20:53:54.070605+0000","last_became_active":"2024-03-23T20:53:52.993514+0000","last_became_peered":"2024-03-23T20:53:52.993514+0000","last_unstale":"2024-03-23T20:53:54.070605+0000","last_undegraded":"2024-03-23T20:53:54.070605+0000","last_fullsized":"2024-03-23T20:53:54.070605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:44:48.653382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069147+0000","last_change":"2024-03-23T20:53:52.992887+0000","last_active":"2024-03-23T20:53:54.069147+0000","last_peered":"2024-03-23T20:53:54.069147+0000","last_clean":"2024-03-23T20:53:54.069147+0000","last_became_active":"2024-03-23T20:53:52.992576+0000","last_became_peered":"2024-03-23T20:53:52.992576+0000","last_unstale":"2024-03-23T20:53:54.069147+0000","last_undegraded":"2024-03-23T20:53:54.069147+0000","last_fullsized":"2024-03-23T20:53:54.069147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:37:04.022275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:52.985583+0000","last_change":"2024-03-23T20:53:48.968024+0000","last_active":"2024-03-23T20:53:52.985583+0000","last_peered":"2024-03-23T20:53:52.985583+0000","last_clean":"2024-03-23T20:53:52.985583+0000","last_became_active":"2024-03-23T20:53:48.967226+0000","last_became_peered":"2024-03-23T20:53:48.967226+0000","last_unstale":"2024-03-23T20:53:52.985583+0000","last_undegraded":"2024-03-23T20:53:52.985583+0000","last_fullsized":"2024-03-23T20:53:52.985583+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_clean_scrub_stamp":"2024-03-23T20:53:47.949768+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:38:34.242650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T20:53:57.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-23T20:53:57.693 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:57.694 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-03-23T20:53:57.710 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-23T20:53:57.412778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82604,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032916,"statfs":{"total":289910292480,"available":289825705984,"internally_reserved":0,"allocated":1581056,"data_stored":1292330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.427752"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069234+0000","last_change":"2024-03-23T20:53:52.996000+0000","last_active":"2024-03-23T20:53:54.069234+0000","last_peered":"2024-03-23T20:53:54.069234+0000","last_clean":"2024-03-23T20:53:54.069234+0000","last_became_active":"2024-03-23T20:53:52.995659+0000","last_became_peered":"2024-03-23T20:53:52.995659+0000","last_unstale":"2024-03-23T20:53:54.069234+0000","last_undegraded":"2024-03-23T20:53:54.069234+0000","last_fullsized":"2024-03-23T20:53:54.069234+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:26.628717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069218+0000","last_change":"2024-03-23T20:53:52.995652+0000","last_active":"2024-03-23T20:53:54.069218+0000","last_peered":"2024-03-23T20:53:54.069218+0000","last_clean":"2024-03-23T20:53:54.069218+0000","last_became_active":"2024-03-23T20:53:52.995094+0000","last_became_peered":"2024-03-23T20:53:52.995094+0000","last_unstale":"2024-03-23T20:53:54.069218+0000","last_undegraded":"2024-03-23T20:53:54.069218+0000","last_fullsized":"2024-03-23T20:53:54.069218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:18.937950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069198+0000","last_change":"2024-03-23T20:53:52.994376+0000","last_active":"2024-03-23T20:53:54.069198+0000","last_peered":"2024-03-23T20:53:54.069198+0000","last_clean":"2024-03-23T20:53:54.069198+0000","last_became_active":"2024-03-23T20:53:52.994087+0000","last_became_peered":"2024-03-23T20:53:52.994087+0000","last_unstale":"2024-03-23T20:53:54.069198+0000","last_undegraded":"2024-03-23T20:53:54.069198+0000","last_fullsized":"2024-03-23T20:53:54.069198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:27.484370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069181+0000","last_change":"2024-03-23T20:53:52.995843+0000","last_active":"2024-03-23T20:53:54.069181+0000","last_peered":"2024-03-23T20:53:54.069181+0000","last_clean":"2024-03-23T20:53:54.069181+0000","last_became_active":"2024-03-23T20:53:52.995018+0000","last_became_peered":"2024-03-23T20:53:52.995018+0000","last_unstale":"2024-03-23T20:53:54.069181+0000","last_undegraded":"2024-03-23T20:53:54.069181+0000","last_fullsized":"2024-03-23T20:53:54.069181+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:02.274898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-23T20:53:53.994334+0000","last_change":"2024-03-23T20:53:52.993885+0000","last_active":"2024-03-23T20:53:53.994334+0000","last_peered":"2024-03-23T20:53:53.994334+0000","last_clean":"2024-03-23T20:53:53.994334+0000","last_became_active":"2024-03-23T20:53:52.993597+0000","last_became_peered":"2024-03-23T20:53:52.993597+0000","last_unstale":"2024-03-23T20:53:53.994334+0000","last_undegraded":"2024-03-23T20:53:53.994334+0000","last_fullsized":"2024-03-23T20:53:53.994334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:16.901421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.070620+0000","last_change":"2024-03-23T20:53:52.994757+0000","last_active":"2024-03-23T20:53:54.070620+0000","last_peered":"2024-03-23T20:53:54.070620+0000","last_clean":"2024-03-23T20:53:54.070620+0000","last_became_active":"2024-03-23T20:53:52.994351+0000","last_became_peered":"2024-03-23T20:53:52.994351+0000","last_unstale":"2024-03-23T20:53:54.070620+0000","last_undegraded":"2024-03-23T20:53:54.070620+0000","last_fullsized":"2024-03-23T20:53:54.070620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:46.671091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.070605+0000","last_change":"2024-03-23T20:53:52.993925+0000","last_active":"2024-03-23T20:53:54.070605+0000","last_peered":"2024-03-23T20:53:54.070605+0000","last_clean":"2024-03-23T20:53:54.070605+0000","last_became_active":"2024-03-23T20:53:52.993514+0000","last_became_peered":"2024-03-23T20:53:52.993514+0000","last_unstale":"2024-03-23T20:53:54.070605+0000","last_undegraded":"2024-03-23T20:53:54.070605+0000","last_fullsized":"2024-03-23T20:53:54.070605+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:44:48.653382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:54.069147+0000","last_change":"2024-03-23T20:53:52.992887+0000","last_active":"2024-03-23T20:53:54.069147+0000","last_peered":"2024-03-23T20:53:54.069147+0000","last_clean":"2024-03-23T20:53:54.069147+0000","last_became_active":"2024-03-23T20:53:52.992576+0000","last_became_peered":"2024-03-23T20:53:52.992576+0000","last_unstale":"2024-03-23T20:53:54.069147+0000","last_undegraded":"2024-03-23T20:53:54.069147+0000","last_fullsized":"2024-03-23T20:53:54.069147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:37:04.022275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-23T20:53:52.985583+0000","last_change":"2024-03-23T20:53:48.968024+0000","last_active":"2024-03-23T20:53:52.985583+0000","last_peered":"2024-03-23T20:53:52.985583+0000","last_clean":"2024-03-23T20:53:52.985583+0000","last_became_active":"2024-03-23T20:53:48.967226+0000","last_became_peered":"2024-03-23T20:53:48.967226+0000","last_unstale":"2024-03-23T20:53:52.985583+0000","last_undegraded":"2024-03-23T20:53:52.985583+0000","last_fullsized":"2024-03-23T20:53:52.985583+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_clean_scrub_stamp":"2024-03-23T20:53:47.949768+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:38:34.242650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T20:53:57.711 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-03-23T20:53:57.711 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-03-23T20:53:57.711 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-03-23T20:53:57.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-03-23T20:53:58.008 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T20:53:58.008 INFO:teuthology.orchestra.run.smithi204.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T20:53:58.025 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-03-23T20:53:58.026 INFO:teuthology.run_tasks:Running task exec... 2024-03-23T20:53:58.035 INFO:teuthology.task.exec:Executing custom commands... 2024-03-23T20:53:58.035 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T20:53:58.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:53:58.398 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.4 2024-03-23T20:53:58.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:53:58.758 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.5 2024-03-23T20:53:58.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:53:59.114 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.6 2024-03-23T20:53:59.132 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:53:59.466 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.4 2024-03-23T20:53:59.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:53:59.830 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.5 2024-03-23T20:53:59.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:00.196 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.6 2024-03-23T20:54:00.214 INFO:teuthology.run_tasks:Running task exec... 2024-03-23T20:54:00.225 INFO:teuthology.task.exec:Executing custom commands... 2024-03-23T20:54:00.225 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T20:54:00.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:00.577 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror 2024-03-23T20:54:00.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:00.939 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.0 2024-03-23T20:54:00.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:01.311 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.1 2024-03-23T20:54:01.328 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:01.702 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.2 2024-03-23T20:54:01.721 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:02.090 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.3 2024-03-23T20:54:02.107 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:02.443 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror 2024-03-23T20:54:02.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:02.800 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.0 2024-03-23T20:54:02.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:03.153 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.1 2024-03-23T20:54:03.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:03.510 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.2 2024-03-23T20:54:03.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-03-23T20:54:03.862 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.3 2024-03-23T20:54:03.880 INFO:teuthology.run_tasks:Running task workunit... 2024-03-23T20:54:03.893 INFO:tasks.workunit:Pulling workunits from ref 82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T20:54:03.893 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-23T20:54:03.894 DEBUG:teuthology.orchestra.run.smithi204:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-03-23T20:54:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T20:54:03.899 INFO:teuthology.orchestra.run.smithi204.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-03-23T20:54:03.899 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-03-23T20:54:03.945 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-03-23T20:54:03.945 DEBUG:teuthology.orchestra.run.smithi204:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-03-23T20:54:03.993 INFO:tasks.workunit:timeout=3h 2024-03-23T20:54:03.994 INFO:tasks.workunit:cleanup=True 2024-03-23T20:54:03.994 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T20:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-03-23T20:54:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Note: switching to '82316b04821a6c2c1d458a881920454bbadb8837'. 2024-03-23T20:54:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:state without impacting any branches by switching back to a branch. 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: git switch -c 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Or undo this operation with: 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: git switch - 2024-03-23T20:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-03-23T20:54:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-23T20:54:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-03-23T20:54:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:HEAD is now at 82316b04821 Merge branch 'wip-64881-reef' of https://github.com/sseshasa/ceph into wip-yuri11-testing-2024-03-21-1324-reef 2024-03-23T20:54:49.517 DEBUG:teuthology.orchestra.run.smithi204:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-03-23T20:54:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-23T20:54:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-03-23T20:54:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-23T20:54:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-23T20:54:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-23T20:54:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-03-23T20:54:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-03-23T20:54:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-23T20:54:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-03-23T20:54:50.626 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-23T20:54:50.626 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-03-23T20:54:50.678 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_journal.sh on cluster1.client.mirror... 2024-03-23T20:54:50.679 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_journal.sh... 2024-03-23T20:54:50.679 DEBUG:teuthology.orchestra.run.smithi204:workunit test rbd/rbd_mirror_journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=82316b04821a6c2c1d458a881920454bbadb8837 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2024-03-23T20:54:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ type xmlstarlet 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ XMLSTARLET=xmlstarlet 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ RBD_MIRROR_INSTANCES=2 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CLUSTER1=cluster1 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CLUSTER2=cluster2 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ PEER_CLUSTER_SUFFIX= 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ POOL=mirror 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ PARENT_POOL=mirror_parent 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ NS1=ns1 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ NS2=ns2 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ TEMPDIR= 2024-03-23T20:54:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ID=mirror 2024-03-23T20:54:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-03-23T20:54:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-03-23T20:54:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_POOL_MODE=pool 2024-03-23T20:54:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_IMAGE_MODE=journal 2024-03-23T20:54:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-03-23T20:54:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS='--id mirror' 2024-03-23T20:54:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ LAST_MIRROR_INSTANCE=1 2024-03-23T20:54:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2024-03-23T20:54:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-03-23T20:54:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-03-23T20:54:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_BIN=. 2024-03-23T20:54:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_SRC=. 2024-03-23T20:54:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' -e CMakeCache.txt ']' 2024-03-23T20:54:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 -gt 0 ']' 2024-03-23T20:54:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup 2024-03-23T20:54:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local c 2024-03-23T20:54:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-03-23T20:54:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_tempdir 2024-03-23T20:54:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:54:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mktemp -d 2024-03-23T20:54:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ TEMPDIR=/tmp/tmp.TE9AI0RDtm 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z 1 ']' 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_pools cluster1 cluster2 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_map_file 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_addr 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local admin_key_file 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:54:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-03-23T20:54:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' already exists 2024-03-23T20:54:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:54:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-03-23T20:54:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' already exists 2024-03-23T20:54:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:54:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 pool init mirror 2024-03-23T20:54:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:54:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-03-23T20:54:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:54:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-03-23T20:54:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-03-23T20:54:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-03-23T20:54:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-03-23T20:54:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-03-23T20:54:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-03-23T20:54:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2024-03-23T20:54:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2024-03-23T20:54:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster1 mirror cluster2 2024-03-23T20:54:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:54:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:54:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:54:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:54:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster2 2024-03-23T20:54:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:54:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2024-03-23T20:54:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-03-23T20:54:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:54:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-03-23T20:54:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-03-23T20:54:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T20:54:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:54:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-03-23T20:54:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=8e975f3e-05a8-4128-866c-64401806f592 2024-03-23T20:54:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:54:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:54:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-03-23T20:54:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-03-23T20:54:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster2 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-03-23T20:54:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T20:54:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:54:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=4252c7df-0a91-4b62-a303-fa018a6ecaba 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_pools cluster2 cluster1 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_map_file 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_addr 2024-03-23T20:54:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local admin_key_file 2024-03-23T20:54:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid 2024-03-23T20:54:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:54:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-03-23T20:54:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' already exists 2024-03-23T20:54:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:54:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-03-23T20:55:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' already exists 2024-03-23T20:55:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:55:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 pool init mirror 2024-03-23T20:55:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:55:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-03-23T20:55:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:55:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-03-23T20:55:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-03-23T20:55:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-03-23T20:55:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-03-23T20:55:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-03-23T20:55:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-03-23T20:55:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2024-03-23T20:55:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2024-03-23T20:55:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster2 mirror cluster1 2024-03-23T20:55:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster1 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:55:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-03-23T20:55:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=e0c35d61-fc57-42db-9d89-abba81a9ca7e 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster1 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2024-03-23T20:55:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-03-23T20:55:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-03-23T20:55:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-03-23T20:55:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T20:55:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:55:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=1a9111e3-508f-4f28-a513-9244d478ffa0 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n octopus ']' 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:55:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd set-require-min-compat-client octopus 2024-03-23T20:55:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:set require_min_compat_client to octopus 2024-03-23T20:55:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T20:55:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd set-require-min-compat-client octopus 2024-03-23T20:55:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:set require_min_compat_client to octopus 2024-03-23T20:55:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: add image and test replay' 2024-03-23T20:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:55:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:55:10 TEST: add image and test replay 2024-03-23T20:55:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:55:10 TEST: add image and test replay 2024-03-23T20:55:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2024-03-23T20:55:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:55:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T20:55:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=test 2024-03-23T20:55:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2024-03-23T20:55:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror test key1 value1 2024-03-23T20:55:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=key1 2024-03-23T20:55:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=value1 2024-03-23T20:55:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key1 value1 2024-03-23T20:55:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror test key2 value2 2024-03-23T20:55:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=key2 2024-03-23T20:55:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=value2 2024-03-23T20:55:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key2 value2 2024-03-23T20:55:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:55:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:55:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:55:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:55:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:55:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:55:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T20:55:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:55:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:55:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:55:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:55:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T20:55:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:55:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:55:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:55:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:55:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 4 2024-03-23T20:55:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:55:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:55:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:55:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:55:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:55:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-03-23T20:55:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T20:55:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T20:55:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T20:55:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T20:55:19.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T20:55:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T20:55:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2272.76 bytes/sec: 8.9 MiB/s 2024-03-23T20:55:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:55:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:55:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:55:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:55:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:55:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:55:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:55:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:55:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:55:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:55:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:55:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:55:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:55:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:55:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:55:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:55:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:55:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:55:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:55:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:55:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:55:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-03-23T20:55:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T20:55:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:55:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:55:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:55:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:55:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-03-23T20:55:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T20:55:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=4, entry_tid=101' -a 'object_number=1, tag_tid=4, entry_tid=101' = 'object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=3, entry_tid=3' '!=' '' 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=3, entry_tid=3' = '' 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:55:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:55:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=4 2024-03-23T20:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T20:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:55:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=3 2024-03-23T20:55:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:55:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:55:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T20:55:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T20:55:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:55:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:55:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=3 2024-03-23T20:55:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 4 = 3 -a 101 -le 3 2024-03-23T20:55:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:55:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:55:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:55:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:55:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:55:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:55:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:55:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:55:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:55:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:55:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:55:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:55:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:55:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:55:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:55:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:55:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:55:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:55:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-03-23T20:55:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=37], [object_number=0, tag_tid=4, entry_tid=36], [object_number=3, tag_tid=4, entry_tid=35], [object_number=2, tag_tid=4, entry_tid=34]]], state=connected] 2024-03-23T20:55:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:55:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:55:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:55:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:55:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-03-23T20:55:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=37], [object_number=0, tag_tid=4, entry_tid=36], [object_number=3, tag_tid=4, entry_tid=35], [object_number=2, tag_tid=4, entry_tid=34]]], state=connected] 2024-03-23T20:55:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=4, entry_tid=37' 2024-03-23T20:55:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=4, entry_tid=101' -a 'object_number=1, tag_tid=4, entry_tid=101' = 'object_number=1, tag_tid=4, entry_tid=37' 2024-03-23T20:55:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=4, entry_tid=37' '!=' 'object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T20:55:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:55:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=4, entry_tid=37' = 'object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T20:55:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=4, entry_tid=37' 2024-03-23T20:55:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:55:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:55:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=4 2024-03-23T20:55:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=37' 2024-03-23T20:55:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:55:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:55:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=4 2024-03-23T20:55:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:55:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:55:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T20:55:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=37' 2024-03-23T20:55:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:55:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:55:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=37 2024-03-23T20:55:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 4 = 4 -a 101 -le 37 2024-03-23T20:55:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:55:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:55:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:55:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:55:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:55:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:55:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:55:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:55:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:55:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:55:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:55:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:55:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:55:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:55:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-03-23T20:55:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-03-23T20:55:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:55:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-03-23T20:55:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-03-23T20:55:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=4, entry_tid=101' -a 'object_number=1, tag_tid=4, entry_tid=101' = 'object_number=1, tag_tid=4, entry_tid=101' 2024-03-23T20:55:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-03-23T20:55:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:55:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-03-23T20:55:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:55:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:55:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:55:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:55:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-03-23T20:55:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:55:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:15' 2024-03-23T20:55:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:55:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:55:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T20:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-03-23T20:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:55:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:15' 2024-03-23T20:55:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:55:15' 2024-03-23T20:55:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:15' 2024-03-23T20:55:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142564263mirror.0smithi20418.2.2-954-g82316b04trueOK42604266mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}42564263mirror.0smithi2042024-03-23 20:55:15' 2024-03-23T20:55:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:15' 2024-03-23T20:55:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:55:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:55:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:55:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-03-23T20:55:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test down+unknown 2024-03-23T20:55:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-03-23T20:55:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test down+unknown '' '' 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:55:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:55:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:55:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:15' 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*down+unknown' 2024-03-23T20:55:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: down+unknown 2024-03-23T20:55:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: status not found 2024-03-23T20:55:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo down+unknown 2024-03-23T20:55:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: ' 2024-03-23T20:55:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update= 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test down+unknown '' 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local prev_last_update= 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='UNKNOWNUNKNOWNOK1testb9e6df0f-082f-4f12-9e35-9ad8fce98a8cdown+unknownstatus not foundcluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}2024-03-23 20:55:15' 2024-03-23T20:55:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:55:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update= 2024-03-23T20:55:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:55:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=down+unknown 2024-03-23T20:55:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo down+unknown 2024-03-23T20:55:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep down+unknown 2024-03-23T20:55:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:down+unknown 2024-03-23T20:55:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T20:55:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T20:55:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T20:55:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T20:55:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:55:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T20:55:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T20:55:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:55:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:55:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:55:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:24.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_meta cluster1 mirror test key1 value1 2024-03-23T20:55:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=key1 2024-03-23T20:55:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local value=value1 2024-03-23T20:55:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key1 2024-03-23T20:55:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test value1 = value1 2024-03-23T20:55:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_meta cluster1 mirror test key2 value2 2024-03-23T20:55:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=key2 2024-03-23T20:55:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local value=value2 2024-03-23T20:55:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key2 2024-03-23T20:55:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test value2 = value2 2024-03-23T20:55:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: stop mirror, add image, start mirror and test replay' 2024-03-23T20:55:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:55:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:55:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:55:24 TEST: stop mirror, add image, start mirror and test replay 2024-03-23T20:55:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:55:24 TEST: stop mirror, add image, start mirror and test replay 2024-03-23T20:55:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster1 2024-03-23T20:55:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T20:55:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:0 '' 2024-03-23T20:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T20:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:0 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:0 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-03-23T20:55:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T20:55:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T20:55:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=29890 2024-03-23T20:55:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 29890 ']' 2024-03-23T20:55:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 29890 2024-03-23T20:55:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T20:55:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T20:55:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=29890 '$2 == pid {print; exit 1}' 2024-03-23T20:55:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:55:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T20:55:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=29890 '$2 == pid {print; exit 1}' 2024-03-23T20:55:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:0 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-03-23T20:55:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T20:55:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T20:55:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:1 '' 2024-03-23T20:55:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T20:55:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T20:55:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-03-23T20:55:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T20:55:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T20:55:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=29918 2024-03-23T20:55:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 29918 ']' 2024-03-23T20:55:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 29918 2024-03-23T20:55:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T20:55:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T20:55:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=29918 '$2 == pid {print; exit 1}' 2024-03-23T20:55:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:55:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T20:55:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=29918 '$2 == pid {print; exit 1}' 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-03-23T20:55:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:55:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:1 2024-03-23T20:55:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-03-23T20:55:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T20:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T20:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image1=test1 2024-03-23T20:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test1 2024-03-23T20:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T20:55:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test1 2024-03-23T20:55:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test1 100 2024-03-23T20:55:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T20:55:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T20:55:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T20:55:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test1 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T20:55:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T20:55:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T20:55:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2500.03 bytes/sec: 9.8 MiB/s 2024-03-23T20:55:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2024-03-23T20:55:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2024-03-23T20:55:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T20:55:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2024-03-23T20:55:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:55:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-03-23T20:55:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:55:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:55:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:27.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:55:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test1'] 2024-03-23T20:55:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1 2024-03-23T20:55:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2024-03-23T20:55:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test1'] 2024-03-23T20:55:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T20:55:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:55:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:55:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test1'] 2024-03-23T20:55:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2024-03-23T20:55:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test1'] 2024-03-23T20:55:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:55:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T20:55:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:55:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:55:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:55:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:55:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test1 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test1 2024-03-23T20:55:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:55:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:55:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:55:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:55:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:55:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:55:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test1 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test1 ']' 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test1' 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test1 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test1 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test1 2024-03-23T20:55:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test1 2024-03-23T20:55:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test1 2024-03-23T20:55:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test1 '' 2024-03-23T20:55:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test1 2024-03-23T20:55:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:55:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test1.status 2024-03-23T20:55:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test1.status 2024-03-23T20:55:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.status 2024-03-23T20:55:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test1 2024-03-23T20:55:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.status 2024-03-23T20:55:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T20:55:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T20:55:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.status 2024-03-23T20:55:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T20:55:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test1 2024-03-23T20:55:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test1 2024-03-23T20:55:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test1 '..*' 2024-03-23T20:55:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test1 2024-03-23T20:55:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:55:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test1.status 2024-03-23T20:55:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test1.status 2024-03-23T20:55:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.status 2024-03-23T20:55:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test1 2024-03-23T20:55:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.status 2024-03-23T20:55:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T20:55:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T20:55:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.status 2024-03-23T20:55:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T20:55:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T20:55:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position 2024-03-23T20:55:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:55:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position '' 2024-03-23T20:55:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:55:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:55:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:55:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:55:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:55:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:55:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:31' 2024-03-23T20:55:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:55:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:55:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T20:55:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:55:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:55:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:31' 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-03-23 20:55:31' 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:31' 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}42934299mirror.1smithi2042024-03-23 20:55:31test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:55:31' 2024-03-23T20:55:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:55:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:31' 2024-03-23T20:55:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:55:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:55:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:55:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test1 down+unknown 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test1 down+unknown '' '' 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test1 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:31' 2024-03-23T20:55:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*down+unknown' 2024-03-23T20:55:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: down+unknown 2024-03-23T20:55:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: status not found 2024-03-23T20:55:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo down+unknown 2024-03-23T20:55:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: ' 2024-03-23T20:55:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update= 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test1 down+unknown '' 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local prev_last_update= 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='UNKNOWNUNKNOWNOK2testb9e6df0f-082f-4f12-9e35-9ad8fce98a8cdown+unknownstatus not foundcluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}2024-03-23 20:55:31test1815fb3b0-78c1-4556-8282-66c7eb9bf58cdown+unknownstatus not foundcluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}2024-03-23 20:55:31' 2024-03-23T20:55:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:55:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update= 2024-03-23T20:55:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:55:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=down+unknown 2024-03-23T20:55:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo down+unknown 2024-03-23T20:55:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep down+unknown 2024-03-23T20:55:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:down+unknown 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test1 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test1.export 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test1.export 2024-03-23T20:55:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.export 2024-03-23T20:55:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test1.export 2024-03-23T20:55:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test1.export 2024-03-23T20:55:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test1.export 2024-03-23T20:55:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test1.export 2024-03-23T20:55:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test1 /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.export 2024-03-23T20:55:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:55:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test1 /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test1.export 2024-03-23T20:55:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:55:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test1.export 2024-03-23T20:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test1.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test1.export 2024-03-23T20:55:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: test the first image is replaying after restart' 2024-03-23T20:55:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:55:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:55:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:55:34 TEST: test the first image is replaying after restart 2024-03-23T20:55:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:55:34 TEST: test the first image is replaying after restart 2024-03-23T20:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-03-23T20:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T20:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T20:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T20:55:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T20:55:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T20:55:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T20:55:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2777.81 bytes/sec: 11 MiB/s 2024-03-23T20:55:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:55:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:55:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:55:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:55:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:55:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:55:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:55:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:55:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:55:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:55:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:55:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:55:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:55:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:55:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:55:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:55:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:55:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:55:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:55:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:55:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:55:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:55:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:55:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush mirror/test1 2024-03-23T20:55:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:55:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2024-03-23T20:55:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:55:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:55:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:55:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:55:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T20:55:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T20:55:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:55:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:55:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:55:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:55:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:55:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T20:55:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T20:55:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:55:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T20:55:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T20:55:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-03-23T20:55:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:55:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-03-23T20:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:55:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:55:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:55:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}} 2024-03-23T20:55:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:55:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:31' 2024-03-23T20:55:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:55:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:55:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T20:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}} 2024-03-23T20:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:55:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:31' 2024-03-23T20:55:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:55:31' 2024-03-23T20:55:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:31' 2024-03-23T20:55:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}42934299mirror.1smithi2042024-03-23 20:55:31test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:55:31' 2024-03-23T20:55:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:55:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:31' 2024-03-23T20:55:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:55:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:55:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:55:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T20:55:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T20:55:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T20:55:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T20:55:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:55:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T20:55:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T20:55:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:55:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:55:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:55:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:55:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:55:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-03-23T20:55:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: stop/start/restart mirror via admin socket' 2024-03-23T20:55:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:55:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:55:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:55:38 TEST: stop/start/restart mirror via admin socket 2024-03-23T20:55:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:55:38 TEST: stop/start/restart mirror via admin socket 2024-03-23T20:55:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2024-03-23T20:55:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:55:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:55:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:55:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2024-03-23T20:55:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:55:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:55:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:55:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:55:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:55:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:55:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:55:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:55:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:55:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:55:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:55:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:55:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:55:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2024-03-23T20:55:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:55:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:55:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:55:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:55:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:55:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:55:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:55:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:55:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:55:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:55:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:55:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:55:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:55:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:55:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:55:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:38' 2024-03-23T20:55:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:55:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:55:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T20:55:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:55:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:55:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:55:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:38' 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:55:38' 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:38' 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+stoppedstopped42934299mirror.1smithi2042024-03-23 20:55:38test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:55:38' 2024-03-23T20:55:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:55:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:38' 2024-03-23T20:55:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:55:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:55:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:55:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:55:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:55:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:55:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:55:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:55:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:55:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:55:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:38' 2024-03-23T20:55:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:55:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:55:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T20:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:55:43.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:55:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:55:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:38' 2024-03-23T20:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-03-23 20:55:38' 2024-03-23T20:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:38' 2024-03-23T20:55:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+stoppedstopped42934299mirror.1smithi2042024-03-23 20:55:38test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:55:38' 2024-03-23T20:55:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:55:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:38' 2024-03-23T20:55:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:55:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:55:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:55:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:55:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:55:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror start 2024-03-23T20:55:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start 2024-03-23T20:55:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start 2024-03-23T20:55:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror start 2024-03-23T20:55:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:55:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start 2024-03-23T20:55:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:55:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:55:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:55:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:55:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:55:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:55:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:55:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:55:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:55:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:55:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:55:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:55:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:55:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-03-23T20:55:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:55:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:55:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:55:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:55:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:55:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:55:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:55:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:55:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:55:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:55:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:55:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:55:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:55:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:55:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:43' 2024-03-23T20:55:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:55:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:55:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:55:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:55:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:43' 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:55:43' 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:43' 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:55:43test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:55:43' 2024-03-23T20:55:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:55:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:43' 2024-03-23T20:55:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:55:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:55:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:55:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-03-23T20:55:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:55:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:55:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:55:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:55:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:55:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:43' 2024-03-23T20:55:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:55:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:55:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:55:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:55:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:43' 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-03-23 20:55:43' 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:43' 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:55:43test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:55:43' 2024-03-23T20:55:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:55:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:43' 2024-03-23T20:55:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:55:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:55:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:55:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2024-03-23T20:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2024-03-23T20:55:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2024-03-23T20:55:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:55:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:55:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2024-03-23T20:55:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:55:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:55:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:55:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:55:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:55:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:55:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:55:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:55:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:55:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:55:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-03-23T20:55:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:55:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:55:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:55:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:55:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:55:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:55:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:55:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:55:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:55:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:55:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:55:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:48' 2024-03-23T20:55:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:55:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:55:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:55:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:55:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:48' 2024-03-23T20:55:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:55:48' 2024-03-23T20:55:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:48' 2024-03-23T20:55:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:55:48test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:55:48' 2024-03-23T20:55:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:55:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:48' 2024-03-23T20:55:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:55:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:55:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:55:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-03-23T20:55:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:55:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:55:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:55:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:55:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:55:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:55:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:48' 2024-03-23T20:55:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:55:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:55:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:55:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:55:53.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:53.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:48' 2024-03-23T20:55:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-03-23 20:55:48' 2024-03-23T20:55:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:55:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:48' 2024-03-23T20:55:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:55:48test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:55:48' 2024-03-23T20:55:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:55:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:48' 2024-03-23T20:55:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:55:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:55:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:55:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:55:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:55:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2024-03-23T20:55:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2024-03-23T20:55:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2024-03-23T20:55:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2024-03-23T20:55:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:55:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:55:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:55:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:55:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:55:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:55:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:55:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:55:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:55:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:55:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:55:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2024-03-23T20:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:55:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:55:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:55:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:55:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:55:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:55:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:55:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:55:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:55:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:55:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:55:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:55:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:55:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:55:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:55:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:55:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:55:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:55:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:55:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:55:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:55:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:55:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:55:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:55:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:55:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:55:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:55:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:53' 2024-03-23T20:55:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:55:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:55:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T20:55:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:55:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:55:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:55:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:53' 2024-03-23T20:55:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:55:53' 2024-03-23T20:55:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:53' 2024-03-23T20:55:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+stoppedstopped42934299mirror.1smithi2042024-03-23 20:55:53test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:55:53' 2024-03-23T20:55:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:55:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:53' 2024-03-23T20:55:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:55:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:55:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:55:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:55:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:55:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2024-03-23T20:55:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:55:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2024-03-23T20:55:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:55:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:55:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:55:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:55:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:55:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:55:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:53' 2024-03-23T20:55:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:55:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:55:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:55:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T20:55:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:55:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:55:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:55:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:55:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:55:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:55:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:55:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:55:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:53' 2024-03-23T20:55:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-03-23 20:55:53' 2024-03-23T20:55:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:55:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:55:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:53' 2024-03-23T20:55:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:55:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:55:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:55:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+stoppedstopped42934299mirror.1smithi2042024-03-23 20:55:53test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:55:53' 2024-03-23T20:55:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:55:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:55:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:53' 2024-03-23T20:55:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:55:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:55:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:55:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:55:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:55:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:55:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2024-03-23T20:55:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:55:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2024-03-23T20:55:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:55:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:55:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:55:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:55:58.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2024-03-23T20:55:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:55:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2024-03-23T20:55:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:55:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:55:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:55:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:55:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:55:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:55:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:55:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2024-03-23T20:55:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:56:00.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:00.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-03-23T20:56:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-03-23T20:56:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:56:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:56:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:01.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:56:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:56:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:56:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:56:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:56:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:59' 2024-03-23T20:56:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:56:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:59' 2024-03-23T20:56:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:55:59' 2024-03-23T20:56:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:59' 2024-03-23T20:56:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:55:59test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:55:58' 2024-03-23T20:56:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:56:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:59' 2024-03-23T20:56:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:56:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-03-23T20:56:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:56:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:56:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:56:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:56:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:56:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:55:58' 2024-03-23T20:56:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:56:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:56:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:58' 2024-03-23T20:56:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-03-23 20:55:58' 2024-03-23T20:56:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:55:58' 2024-03-23T20:56:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:55:59test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:55:58' 2024-03-23T20:56:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:56:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:55:58' 2024-03-23T20:56:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:56:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2 2024-03-23T20:56:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2 2024-03-23T20:56:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2 2024-03-23T20:56:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2 2024-03-23T20:56:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2 2024-03-23T20:56:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:56:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:56:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:56:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:56:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:56:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:56:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:05.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:56:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:56:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:56:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2024-03-23T20:56:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:56:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:06.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:56:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:56:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:56:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:56:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:56:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:56:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:56:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:56:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:56:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:56:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:56:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:56:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:04' 2024-03-23T20:56:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:56:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:56:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T20:56:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:56:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:56:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:56:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:04' 2024-03-23T20:56:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:56:04' 2024-03-23T20:56:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:04' 2024-03-23T20:56:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+stoppedstopped42934299mirror.1smithi2042024-03-23 20:56:04test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:56:03' 2024-03-23T20:56:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:56:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:04' 2024-03-23T20:56:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:56:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:56:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:56:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:56:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:56:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2024-03-23T20:56:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2024-03-23T20:56:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:56:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:56:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:56:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:56:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:56:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:56:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:03' 2024-03-23T20:56:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:56:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:56:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T20:56:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:56:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:56:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:56:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:03' 2024-03-23T20:56:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-03-23 20:56:03' 2024-03-23T20:56:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:03' 2024-03-23T20:56:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+stoppedstopped42934299mirror.1smithi2042024-03-23 20:56:04test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:56:03' 2024-03-23T20:56:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:56:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:03' 2024-03-23T20:56:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:56:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:56:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:56:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:56:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:56:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2024-03-23T20:56:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror start mirror/test1 2024-03-23T20:56:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror/test 2024-03-23T20:56:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:09.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror/test 2024-03-23T20:56:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:56:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:56:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:56:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:56:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:56:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:56:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:56:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:56:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:09' 2024-03-23T20:56:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:56:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:09' 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:56:09' 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:09' 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:56:09test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:56:03' 2024-03-23T20:56:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:56:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:09' 2024-03-23T20:56:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:56:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror start mirror cluster2 2024-03-23T20:56:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror cluster2 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror cluster2 2024-03-23T20:56:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror cluster2 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror cluster2 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-03-23T20:56:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:56:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:56:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:56:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-03-23T20:56:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:56:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:56:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:56:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:56:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:56:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:11' 2024-03-23T20:56:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:56:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:56:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:11' 2024-03-23T20:56:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-03-23 20:56:11' 2024-03-23T20:56:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:11' 2024-03-23T20:56:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:56:09test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:56:11' 2024-03-23T20:56:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:56:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:11' 2024-03-23T20:56:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:56:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test 2024-03-23T20:56:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:56:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:56:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:56:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test 2024-03-23T20:56:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror restart mirror/test1 2024-03-23T20:56:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror/test 2024-03-23T20:56:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror/test 2024-03-23T20:56:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:56:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:56:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:56:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:56:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:56:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:56:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:14' 2024-03-23T20:56:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:56:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:14' 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:56:14' 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:14' 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:56:14test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:56:11' 2024-03-23T20:56:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:56:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:14' 2024-03-23T20:56:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:56:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2 2024-03-23T20:56:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2 2024-03-23T20:56:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2 2024-03-23T20:56:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:56:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:56:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-03-23T20:56:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:56:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2 2024-03-23T20:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2 2024-03-23T20:56:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:56:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:56:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:56:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:56:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:56:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:56:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2024-03-23T20:56:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2024-03-23T20:56:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:56:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2024-03-23T20:56:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:56:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:56:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:56:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:56:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:56:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:56:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:56:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:56:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:56:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:56:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:56:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:20' 2024-03-23T20:56:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:56:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:56:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T20:56:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:56:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:56:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:56:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:20' 2024-03-23T20:56:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:56:20' 2024-03-23T20:56:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:20' 2024-03-23T20:56:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+stoppedstopped42934299mirror.1smithi2042024-03-23 20:56:20test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:56:19' 2024-03-23T20:56:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:56:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:20' 2024-03-23T20:56:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:56:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:56:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:56:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:56:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:56:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2024-03-23T20:56:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:56:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:56:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:56:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:56:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:56:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:56:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:19' 2024-03-23T20:56:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:56:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:56:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T20:56:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:56:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:56:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:56:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:19' 2024-03-23T20:56:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-03-23 20:56:19' 2024-03-23T20:56:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:19' 2024-03-23T20:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+stoppedstopped42934299mirror.1smithi2042024-03-23 20:56:20test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+stoppedstopped42894296mirror.0smithi2042024-03-23 20:56:19' 2024-03-23T20:56:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:56:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:19' 2024-03-23T20:56:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:56:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:56:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2 2024-03-23T20:56:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2 2024-03-23T20:56:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2 2024-03-23T20:56:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2 2024-03-23T20:56:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:56:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:56:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test1 2024-03-23T20:56:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-03-23T20:56:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-03-23T20:56:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-03-23T20:56:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-03-23T20:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-03-23T20:56:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:56:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:26' 2024-03-23T20:56:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:56:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:26' 2024-03-23T20:56:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:56:26' 2024-03-23T20:56:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:26' 2024-03-23T20:56:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:56:26test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:56:26' 2024-03-23T20:56:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:56:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:26' 2024-03-23T20:56:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:56:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-03-23T20:56:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-03-23T20:56:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-03-23T20:56:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 815fb3b0-78c1-4556-8282-66c7eb9bf58c 2024-03-23T20:56:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:56:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:56:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:26' 2024-03-23T20:56:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T20:56:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:56:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:26' 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-03-23 20:56:26' 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:26' 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtestb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:56:26test1815fb3b0-78c1-4556-8282-66c7eb9bf58cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:56:26' 2024-03-23T20:56:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-03-23T20:56:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:26' 2024-03-23T20:56:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-03-23T20:56:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool= 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image= 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush 2024-03-23T20:56:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:56:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:56:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:56:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush 2024-03-23T20:56:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror status 2024-03-23T20:56:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror status 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:{ 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool_replayers": [ 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "peer": "uuid: 8e975f3e-05a8-4128-866c-64401806f592 cluster: cluster2 client: client.mirror", 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool": "mirror", 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instance_id": "4296", 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "stopped (manual)", 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader_instance_id": "4296", 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader": true, 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instances": [ 2024-03-23T20:56:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "4296", 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "4299" 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ], 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.30783.cluster1.94066872408216.asok", 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.30783.cluster2.94066885179544.asok", 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "sync_throttler": { 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 5, 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "deletion_throttler": { 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 1, 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [ 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "mirror/test1", 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ], 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter": { 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter_status": { 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "delete_images_queue": [], 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "failed_deletes_queue": [] 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "namespaces": [ 2024-03-23T20:56:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "ns1", 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter": { 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter_status": { 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "delete_images_queue": [], 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "failed_deletes_queue": [] 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "ns2", 2024-03-23T20:56:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter": { 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter_status": { 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "delete_images_queue": [], 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "failed_deletes_queue": [] 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ] 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "peer": "uuid: 4252c7df-0a91-4b62-a303-fa018a6ecaba cluster: cluster2 client: client.mirror", 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool": "mirror_parent", 2024-03-23T20:56:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instance_id": "4302", 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "stopped (manual)", 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader_instance_id": "4302", 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader": true, 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instances": [ 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "4302", 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "4305" 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ], 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.30783.cluster1.94066872809624.asok", 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.30783.cluster2.94066874644632.asok", 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "sync_throttler": { 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 5, 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "deletion_throttler": { 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 1, 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter": { 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter_status": { 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "delete_images_queue": [], 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "failed_deletes_queue": [] 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "namespaces": [] 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ] 2024-03-23T20:56:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:} 2024-03-23T20:56:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror status 2024-03-23T20:56:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status 2024-03-23T20:56:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:{ 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool_replayers": [ 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "peer": "uuid: 8e975f3e-05a8-4128-866c-64401806f592 cluster: cluster2 client: client.mirror", 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool": "mirror", 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instance_id": "4299", 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "stopped (manual)", 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader_instance_id": "4296", 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader": false, 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.30811.cluster1.94673415559320.asok", 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.30811.cluster2.94673428240536.asok", 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "sync_throttler": { 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 5, 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-03-23T20:56:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "deletion_throttler": { 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 1, 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [ 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "mirror/test", 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ], 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "namespaces": [ 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "ns1", 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [] 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "ns2", 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [] 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ] 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "peer": "uuid: 4252c7df-0a91-4b62-a303-fa018a6ecaba cluster: cluster2 client: client.mirror", 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool": "mirror_parent", 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instance_id": "4305", 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "stopped (manual)", 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader_instance_id": "4302", 2024-03-23T20:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader": false, 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.30811.cluster1.94673415870616.asok", 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.30811.cluster2.94673417672856.asok", 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "sync_throttler": { 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 5, 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "deletion_throttler": { 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 1, 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "namespaces": [] 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-03-23T20:56:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ] 2024-03-23T20:56:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:} 2024-03-23T20:56:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test1 2024-03-23T20:56:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T20:56:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test1 2024-03-23T20:56:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-03-23T20:56:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test1 2024-03-23T20:56:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test1 2024-03-23T20:56:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T20:56:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: test image rename' 2024-03-23T20:56:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:56:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:56:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:56:30 TEST: test image rename 2024-03-23T20:56:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:56:30 TEST: test image rename 2024-03-23T20:56:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ new_name=test_RENAMED 2024-03-23T20:56:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_image cluster2 mirror test test_RENAMED 2024-03-23T20:56:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_name=test_RENAMED 2024-03-23T20:56:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rename mirror/test mirror/test_RENAMED 2024-03-23T20:56:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2024-03-23T20:56:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2024-03-23T20:56:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2024-03-23T20:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2024-03-23T20:56:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2024-03-23T20:56:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T20:56:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2024-03-23T20:56:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:33.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:33.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2024-03-23T20:56:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' '' 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2024-03-23T20:56:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_RENAMED: 2024-03-23T20:56:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:56:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:56:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:26' 2024-03-23T20:56:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-03-23T20:56:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:56:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:26' 2024-03-23T20:56:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2024-03-23 20:56:26' 2024-03-23T20:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:26' 2024-03-23T20:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtest_RENAMEDb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42934299mirror.1smithi2042024-03-23 20:56:26' 2024-03-23T20:56:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update' 2024-03-23T20:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:26' 2024-03-23T20:56:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state' 2024-03-23T20:56:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2024-03-23T20:56:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:{ 2024-03-23T20:56:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "mirror/test_RENAMED", 2024-03-23T20:56:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:} 2024-03-23T20:56:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test_RENAMED 2024-03-23T20:56:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:56:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:56:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:56:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test_RENAMED 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test_RENAMED 2024-03-23T20:56:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2024-03-23T20:56:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror restart : restart rbd mirror 2024-03-23T20:56:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror/test_RENAMED 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T20:56:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror/test_RENAMED 2024-03-23T20:56:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2024-03-23T20:56:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2024-03-23T20:56:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2024-03-23T20:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' '' 2024-03-23T20:56:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2024-03-23T20:56:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_RENAMED: 2024-03-23T20:56:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: b9e6df0f-082f-4f12-9e35-9ad8fce98a8c 2024-03-23T20:56:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2024-03-23T20:56:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T20:56:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:34' 2024-03-23T20:56:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2024-03-23T20:56:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T20:56:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:34' 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2024-03-23 20:56:34' 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:34' 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtest_RENAMEDb9e6df0f-082f-4f12-9e35-9ad8fce98a8cup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}}42934299mirror.1smithi2042024-03-23 20:56:34' 2024-03-23T20:56:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update' 2024-03-23T20:56:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:34' 2024-03-23T20:56:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state' 2024-03-23T20:56:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_image cluster2 mirror test_RENAMED test 2024-03-23T20:56:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-03-23T20:56:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_name=test 2024-03-23T20:56:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rename mirror/test_RENAMED mirror/test 2024-03-23T20:56:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:56:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:56:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:38.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-03-23T20:56:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test_RENAMED 2024-03-23T20:56:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T20:56:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: test trash move restore' 2024-03-23T20:56:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:56:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:56:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:56:40 TEST: test trash move restore 2024-03-23T20:56:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:56:40 TEST: test trash move restore 2024-03-23T20:56:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster2 mirror test 2024-03-23T20:56:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster2 info mirror/test 2024-03-23T20:56:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T20:56:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10a966cb621b 2024-03-23T20:56:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ trash_move cluster2 mirror test 2024-03-23T20:56:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 trash move mirror/test 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:56:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T20:56:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T20:56:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T20:56:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10a7334fe012 2024-03-23T20:56:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T20:56:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T20:56:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10a7334fe012 2024-03-23T20:56:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T20:56:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10a7334fe012 2024-03-23T20:56:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T20:56:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T20:56:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T20:56:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T20:56:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T20:56:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T20:56:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T20:56:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:56:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T20:56:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ trash_restore cluster2 mirror 10a966cb621b 2024-03-23T20:56:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10a966cb621b 2024-03-23T20:56:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 trash restore mirror/10a966cb621b 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:56:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T20:56:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T20:56:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed (with rbd-mirror on one cluster)' 2024-03-23T20:56:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:56:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:56:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:56:45 TEST: check if removed 'images'\''' OMAP are removed '(with' rbd-mirror on one 'cluster)' 2024-03-23T20:56:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:56:45 TEST: check if removed images' OMAP are removed (with rbd-mirror on one cluster) 2024-03-23T20:56:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test 2024-03-23T20:56:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T20:56:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test 2024-03-23T20:56:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-03-23T20:56:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-03-23T20:56:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T20:56:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=status_global 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-03-23T20:56:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-03-23T20:56:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-03-23T20:56:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-03-23T20:56:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-03-23T20:56:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-03-23T20:56:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-03-23T20:56:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-03-23T20:56:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-03-23T20:56:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-03-23T20:56:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T20:56:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:56:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T20:56:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-03-23T20:56:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-03-23T20:56:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_ 2024-03-23T20:56:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-03-23T20:56:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_ 2024-03-23T20:56:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-03-23T20:56:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_ 2024-03-23T20:56:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-03-23T20:56:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirror_leader 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_map 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-03-23T20:56:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T20:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-03-23T20:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-03-23T20:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-03-23T20:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T20:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-03-23T20:56:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-03-23T20:56:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-03-23T20:56:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-03-23T20:56:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-03-23T20:56:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T20:56:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-03-23T20:56:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-03-23T20:56:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-03-23T20:56:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T20:56:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-03-23T20:56:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-03-23T20:56:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-03-23T20:56:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-03-23T20:56:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:56:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-03-23T20:56:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T20:56:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-03-23T20:56:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-03-23T20:56:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-03-23T20:56:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T20:56:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-03-23T20:56:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T20:56:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=status_global 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-03-23T20:56:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-03-23T20:56:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_ 2024-03-23T20:56:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_ 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_ 2024-03-23T20:56:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-03-23T20:56:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirror_leader 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_map 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T20:56:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-03-23T20:56:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-03-23T20:56:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-03-23T20:56:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error getting omap key set mirror/rbd_mirror_leader: (2) No such file or directory 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror test 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=journal 2024-03-23T20:56:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2024-03-23T20:56:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' pool = image ']' 2024-03-23T20:56:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' mirror = mirror_parent ']' 2024-03-23T20:56:49.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:56:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:56:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T20:56:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T20:56:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T20:56:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T20:56:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T20:56:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T20:56:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T20:56:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T20:56:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T20:56:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T20:56:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T20:56:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T20:56:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T20:56:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T20:56:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T20:56:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T20:56:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T20:56:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:56:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-03-23T20:56:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T20:56:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T20:56:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:56:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T20:56:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T20:56:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T20:56:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T20:56:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2500.03 bytes/sec: 9.8 MiB/s 2024-03-23T20:56:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:56:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:56:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:56:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:56:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:56:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:56:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:56:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:56:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:56:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:56:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:56:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:56:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:56:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:56:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:56:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:56:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:56:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:56:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T20:56:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:56:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:56:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:56:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:56:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:56:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:56:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:56:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T20:56:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:56:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:56:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-03-23T20:56:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T20:56:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:56:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:56:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-03-23T20:56:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:56:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:56:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T20:56:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T20:56:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:56:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:56:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=3 2024-03-23T20:56:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 1 -a 101 -le 3 2024-03-23T20:56:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:56:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:56:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:56:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:56:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:56:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:56:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:56:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:56:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:56:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:56:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:56:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:56:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=47], [object_number=2, tag_tid=2, entry_tid=46], [object_number=1, tag_tid=2, entry_tid=45], [object_number=0, tag_tid=2, entry_tid=44]]], state=connected] 2024-03-23T20:56:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:56:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:56:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:56:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:56:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:56:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:56:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:56:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=47], [object_number=2, tag_tid=2, entry_tid=46], [object_number=1, tag_tid=2, entry_tid=45], [object_number=0, tag_tid=2, entry_tid=44]]], state=connected] 2024-03-23T20:56:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=47' 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=2, entry_tid=47' 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=2, entry_tid=47' '!=' 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=2, entry_tid=47' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=3, tag_tid=2, entry_tid=47' 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:56:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:56:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-03-23T20:56:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=47' 2024-03-23T20:56:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:56:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:56:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=2 2024-03-23T20:56:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:56:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:56:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T20:56:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=47' 2024-03-23T20:56:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:56:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:56:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=47 2024-03-23T20:56:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 -a 101 -le 47 2024-03-23T20:56:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:56:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:56:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:56:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:56:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:56:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:56:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:56:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:56:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:56:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:56:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:56:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:56:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:56:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:56:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:56:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:56:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:56:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:56:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:56:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:56:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:56:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:56:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:56:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:56:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:56:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:56:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:56:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:56:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:56:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:56:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:56:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:56:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":175.0,"entries_behind_primary":0,"entries_per_second":2.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2}} 2024-03-23T20:56:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:56:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:56' 2024-03-23T20:56:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:56:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:56:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:56:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":175.0,"entries_behind_primary":0,"entries_per_second":2.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2}} 2024-03-23T20:56:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:56:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:56:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:56:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:56:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:56:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:56:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:56' 2024-03-23T20:56:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:56:56' 2024-03-23T20:56:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:56:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:56:56' 2024-03-23T20:56:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:56:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:56:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:56:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":175.0,"entries_behind_primary":0,"entries_per_second":2.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2}}42894296mirror.0smithi2042024-03-23 20:56:56' 2024-03-23T20:56:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:56:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:56:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:56:56' 2024-03-23T20:56:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:56:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:56:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:56:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:56:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:56:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:56:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: failover and failback' 2024-03-23T20:56:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:56:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:56:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:56:57 TEST: failover and failback 2024-03-23T20:56:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:56:57 TEST: failover and failback 2024-03-23T20:56:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster2 2024-03-23T20:56:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:56:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:0 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:56:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:56:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:56:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:56:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T20:56:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:1 2024-03-23T20:56:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T20:56:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:56:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T20:56:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T20:56:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:56:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:56:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T20:56:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T20:56:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:56:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T20:56:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:56:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:56:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:56:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:56:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:56:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:56:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:56:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:56:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:56:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:56:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:56:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:56:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:56:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:56:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:56:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:56:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:56:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:56:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:56:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:56:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:56:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:56:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:56:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:56:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:56:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:56:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:56:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:56:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:56:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:56:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:56:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:56:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:56:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:56:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:56:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:56:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:56:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:56:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:56:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:56:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:56:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:57:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:57:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:57:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:57:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:58 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:56:59' 2024-03-23T20:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T20:57:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:57:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:01 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:02' 2024-03-23T20:57:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:01' 2024-03-23T20:57:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:57:01' 2024-03-23T20:57:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:01' 2024-03-23T20:57:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:57:01cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:57:02' 2024-03-23T20:57:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:01' 2024-03-23T20:57:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:57:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:02 2024-03-23T20:57:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:57:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:01' 2024-03-23T20:57:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:02' 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:57:02' 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:02' 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:57:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:57:01' 2024-03-23T20:57:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:02' 2024-03-23T20:57:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T20:57:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:57:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T20:57:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:57:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:57:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:57:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:57:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:57:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:57:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:57:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:57:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:57:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:57:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:57:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:57:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-03-23T20:57:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T20:57:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T20:57:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:57:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T20:57:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T20:57:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T20:57:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T20:57:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2272.75 bytes/sec: 8.9 MiB/s 2024-03-23T20:57:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:57:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:57:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:57:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:57:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:57:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:57:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:57:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:57:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:57:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:57:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:57:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:57:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:57:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:57:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-03-23T20:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:57:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' '!=' '' 2024-03-23T20:57:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:57:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' = '' 2024-03-23T20:57:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=6, entry_tid=101' 2024-03-23T20:57:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:57:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:57:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=6 2024-03-23T20:57:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:57:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:57:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=5 2024-03-23T20:57:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=6, entry_tid=101' 2024-03-23T20:57:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:57:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:57:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T20:57:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:57:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:57:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-03-23T20:57:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 6 = 5 -a 101 -le 0 2024-03-23T20:57:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:57:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:57:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:57:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:57:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:57:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:57:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:57:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:57:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:57:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' '!=' 'object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-03-23T20:57:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:57:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:57:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:57:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:57:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:57:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:57:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-03-23T20:57:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:57:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:57:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T20:57:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' '!=' 'object_number=0, tag_tid=5, entry_tid=0' 2024-03-23T20:57:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2024-03-23T20:57:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:57:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:57:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-03-23T20:57:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=1, tag_tid=6, entry_tid=101' 2024-03-23T20:57:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T20:57:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:57:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:07 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:07' 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:57:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:57:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:57:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:57:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:57:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:07' 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 20:57:07' 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:07' 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 20:57:12cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}2024-03-23 20:57:07' 2024-03-23T20:57:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:12' 2024-03-23T20:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:57:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:57:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:57:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:57:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-03-23T20:57:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:57:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-03-23T20:57:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:57:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:07 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:12' 2024-03-23T20:57:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:57:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:57:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T20:57:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2024-03-23T20:57:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:57:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:57:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:07' 2024-03-23T20:57:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:57:07' 2024-03-23T20:57:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:07' 2024-03-23T20:57:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}42894296mirror.0smithi2042024-03-23 20:57:07cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 20:57:12' 2024-03-23T20:57:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:07' 2024-03-23T20:57:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:57:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:57:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:57:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T20:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T20:57:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T20:57:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T20:57:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:57:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T20:57:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T20:57:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:57:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:57:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:57:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T20:57:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T20:57:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:57:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:57:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:57:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:57:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:57:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:57:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:57:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:57:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:57:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:57:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:57:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:16 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:17' 2024-03-23T20:57:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:16' 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:57:16' 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:16' 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:57:16cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:57:17' 2024-03-23T20:57:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:16' 2024-03-23T20:57:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:57:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:57:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:57:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:17 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:16' 2024-03-23T20:57:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:17' 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:57:17' 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:17' 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:57:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:57:16' 2024-03-23T20:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:17' 2024-03-23T20:57:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T20:57:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:57:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T20:57:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:57:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:57:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:57:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:57:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:57:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:57:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:57:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:57:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:57:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:57:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:57:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:57:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:57:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:57:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:57:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:57:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T20:57:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T20:57:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:57:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T20:57:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:57:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:57:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:57:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:57:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:57:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:57:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:57:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:57:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:57:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:57:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:57:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:57:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:57:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:57:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:57:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:57:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:26 2024-03-23T20:57:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:27' 2024-03-23T20:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:26' 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:57:26' 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:26' 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:57:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:57:27' 2024-03-23T20:57:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:26' 2024-03-23T20:57:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:57:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:27 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:26' 2024-03-23T20:57:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:27' 2024-03-23T20:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:57:27' 2024-03-23T20:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:27' 2024-03-23T20:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:57:27cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:57:26' 2024-03-23T20:57:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:27' 2024-03-23T20:57:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T20:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T20:57:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:57:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:57:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:57:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:57:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:57:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:57:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:57:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:57:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:57:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:57:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:57:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:57:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:57:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:57:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:57:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:57:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=11, entry_tid=0' 2024-03-23T20:57:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:57:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:57:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:57:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-03-23T20:57:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=11, entry_tid=0' 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=11, entry_tid=0' -a 'object_number=0, tag_tid=11, entry_tid=0' = 'object_number=0, tag_tid=11, entry_tid=0' 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-03-23T20:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:57:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2024-03-23T20:57:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:32 2024-03-23T20:57:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:57:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:57:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:32' 2024-03-23T20:57:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:57:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:57:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T20:57:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2024-03-23T20:57:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:57:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:57:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:32' 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:57:32' 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:32' 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}42894296mirror.0smithi2042024-03-23 20:57:32cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 20:57:32' 2024-03-23T20:57:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:32' 2024-03-23T20:57:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:57:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:57:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:57:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:57:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T20:57:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T20:57:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:57:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:57:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:57:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:32 2024-03-23T20:57:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:57:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:57:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2024-03-23T20:57:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:32' 2024-03-23T20:57:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:57:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:57:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:57:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:57:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:57:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:32' 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 20:57:32' 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:32' 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 20:57:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}2024-03-23 20:57:32' 2024-03-23T20:57:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:32' 2024-03-23T20:57:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:57:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:57:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:57:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T20:57:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:57:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T20:57:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T20:57:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:57:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:57:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:57:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T20:57:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T20:57:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:57:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:57:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:57:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:57:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:57:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:57:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:57:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:57:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:57:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:57:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:57:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:37 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:37' 2024-03-23T20:57:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T20:57:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:41 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:42' 2024-03-23T20:57:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:41' 2024-03-23T20:57:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:57:41' 2024-03-23T20:57:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:41' 2024-03-23T20:57:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:57:41cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:57:42' 2024-03-23T20:57:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:41' 2024-03-23T20:57:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:57:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:57:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:42 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:41' 2024-03-23T20:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:42' 2024-03-23T20:57:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:57:42' 2024-03-23T20:57:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:42' 2024-03-23T20:57:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:57:42cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:57:41' 2024-03-23T20:57:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:42' 2024-03-23T20:57:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T20:57:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:57:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T20:57:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:57:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:57:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:57:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:57:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:57:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:57:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:57:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:57:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:57:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:57:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:57:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:57:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster1 mirror test 100 2024-03-23T20:57:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T20:57:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T20:57:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:57:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T20:57:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T20:57:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T20:57:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T20:57:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1315.8 bytes/sec: 5.1 MiB/s 2024-03-23T20:57:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:57:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:57:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:57:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:57:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:57:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:57:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:57:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:57:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:57:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T20:57:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:57:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T20:57:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T20:57:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T20:57:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T20:57:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:57:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2024-03-23T20:57:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-03-23T20:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T20:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T20:57:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:57:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2024-03-23T20:57:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos= 2024-03-23T20:57:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = '' 2024-03-23T20:57:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test '' '!=' '' 2024-03-23T20:57:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T20:57:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T20:57:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T20:57:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T20:57:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T20:57:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T20:57:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:57:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2024-03-23T20:57:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:57:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2024-03-23T20:57:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos= 2024-03-23T20:57:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = '' 2024-03-23T20:57:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test '' '!=' '' 2024-03-23T20:57:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2024-03-23T20:57:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T20:57:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:57:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:57:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T20:57:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T20:57:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:57:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=2, tag_tid=13, entry_tid=86], [object_number=1, tag_tid=13, entry_tid=85], [object_number=0, tag_tid=13, entry_tid=84], [object_number=3, tag_tid=13, entry_tid=83]]], state=connected] 2024-03-23T20:57:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-03-23T20:57:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T20:57:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:57:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=2, tag_tid=13, entry_tid=86], [object_number=1, tag_tid=13, entry_tid=85], [object_number=0, tag_tid=13, entry_tid=84], [object_number=3, tag_tid=13, entry_tid=83]]], state=connected] 2024-03-23T20:57:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=2, tag_tid=13, entry_tid=86' 2024-03-23T20:57:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = 'object_number=2, tag_tid=13, entry_tid=86' 2024-03-23T20:57:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=2, tag_tid=13, entry_tid=86' '!=' '' 2024-03-23T20:57:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:57:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=2, tag_tid=13, entry_tid=86' = '' 2024-03-23T20:57:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=2, tag_tid=13, entry_tid=86' 2024-03-23T20:57:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=13, entry_tid=101' 2024-03-23T20:57:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:57:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:57:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=13 2024-03-23T20:57:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=2, tag_tid=13, entry_tid=86' 2024-03-23T20:57:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:57:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:57:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=13 2024-03-23T20:57:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=13, entry_tid=101' 2024-03-23T20:57:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:57:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:57:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T20:57:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=2, tag_tid=13, entry_tid=86' 2024-03-23T20:57:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:57:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:57:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=86 2024-03-23T20:57:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 13 = 13 -a 101 -le 86 2024-03-23T20:57:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:57:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:57:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:57:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T20:57:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:57:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:57:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T20:57:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T20:57:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:57:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T20:57:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T20:57:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:57:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:57:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-03-23T20:57:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:57:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:57:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:57:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:57:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:57:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:57:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:57:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-03-23T20:57:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = 'object_number=1, tag_tid=13, entry_tid=101' 2024-03-23T20:57:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:57:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:46 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:47' 2024-03-23T20:57:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:57:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:57:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:57:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:57:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:57:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:46' 2024-03-23T20:57:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:57:46' 2024-03-23T20:57:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:57:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:46' 2024-03-23T20:57:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 20:57:46cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}2024-03-23 20:57:47' 2024-03-23T20:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:46' 2024-03-23T20:57:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:57:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:57:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:57:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying primary_position 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying primary_position '' 2024-03-23T20:57:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:57:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:47 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:51' 2024-03-23T20:57:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:57:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:57:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T20:57:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2024-03-23T20:57:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:57:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:57:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:47' 2024-03-23T20:57:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 20:57:47' 2024-03-23T20:57:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:57:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:47' 2024-03-23T20:57:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}44564463mirror.0smithi2042024-03-23 20:57:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 20:57:51' 2024-03-23T20:57:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:47' 2024-03-23T20:57:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:57:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:57:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:57:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:57:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T20:57:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T20:57:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T20:57:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T20:57:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:57:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T20:57:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T20:57:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:57:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:57:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:57:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:57:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:57:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T20:57:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T20:57:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:57:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T20:57:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T20:57:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:57:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:57:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:57:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:57:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:57:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:57:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:57:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:57:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:57:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:57:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:57:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:57:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:57:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:57:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:57:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:57:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:57:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:57:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:57:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:57:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:57:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:57:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:57:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:57:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:57:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:57:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:57:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:57:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:57:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:57:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:56 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:57' 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:56' 2024-03-23T20:57:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:57:56' 2024-03-23T20:57:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:56' 2024-03-23T20:57:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:57:56cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:57:57' 2024-03-23T20:57:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:56' 2024-03-23T20:57:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:57:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:57 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:57:56' 2024-03-23T20:57:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:57:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:57:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:57:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:57:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:57:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:57:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:57:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:57' 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:57:57' 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:57:57' 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:57:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:57:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:57:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:57:57cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:57:56' 2024-03-23T20:57:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:57:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:57:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:57:57' 2024-03-23T20:57:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:57:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:57:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:57:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:57:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:57:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:57:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T20:57:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:57:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:57:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T20:57:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:57:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:57:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:57:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:57:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:57:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:57:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:57:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:57:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:57:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:57:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:57:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:57:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:57:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:57:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:57:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:57:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:57:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:57:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:57:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:57:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:57:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:57:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:57:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:57:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:57:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:57:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:57:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:57:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:57:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:57:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:57:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:58:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:58:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:58:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-03-23T20:58:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T20:58:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T20:58:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T20:58:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T20:58:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T20:58:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T20:58:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T20:58:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2272.75 bytes/sec: 8.9 MiB/s 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:58:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:58:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:58:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:58:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:58:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:58:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:58:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:58:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:58:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:58:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:58:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=18, entry_tid=95], [object_number=2, tag_tid=18, entry_tid=94], [object_number=1, tag_tid=18, entry_tid=93], [object_number=0, tag_tid=18, entry_tid=92]]], state=connected] 2024-03-23T20:58:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:58:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:58:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=18, entry_tid=95], [object_number=2, tag_tid=18, entry_tid=94], [object_number=1, tag_tid=18, entry_tid=93], [object_number=0, tag_tid=18, entry_tid=92]]], state=connected] 2024-03-23T20:58:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=18, entry_tid=95' 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=3, tag_tid=18, entry_tid=95' 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=18, entry_tid=95' '!=' '' 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=18, entry_tid=95' = '' 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=3, tag_tid=18, entry_tid=95' 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=18, entry_tid=101' 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:58:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:58:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=18 2024-03-23T20:58:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=18, entry_tid=95' 2024-03-23T20:58:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T20:58:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:58:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=18 2024-03-23T20:58:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=18, entry_tid=101' 2024-03-23T20:58:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:58:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:58:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T20:58:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=18, entry_tid=95' 2024-03-23T20:58:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T20:58:02.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T20:58:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=95 2024-03-23T20:58:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 18 = 18 -a 101 -le 95 2024-03-23T20:58:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:58:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:58:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:58:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:58:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:58:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:58:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:58:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:58:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:58:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:58:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:58:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:58:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:58:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:58:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:58:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:58:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2024-03-23T20:58:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:58:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:58:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:58:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:58:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=18, entry_tid=101' 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=1, tag_tid=18, entry_tid=101' 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-03-23T20:58:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T20:58:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:02 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:02' 2024-03-23T20:58:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:58:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:58:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T20:58:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2024-03-23T20:58:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:02' 2024-03-23T20:58:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:58:02' 2024-03-23T20:58:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:02' 2024-03-23T20:58:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}42894296mirror.0smithi2042024-03-23 20:58:02cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 20:58:02' 2024-03-23T20:58:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:02' 2024-03-23T20:58:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:58:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:02 2024-03-23T20:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2024-03-23T20:58:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:02' 2024-03-23T20:58:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:58:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:58:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:58:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:02' 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 20:58:02' 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:02' 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 20:58:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}2024-03-23 20:58:02' 2024-03-23T20:58:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:02' 2024-03-23T20:58:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:58:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:58:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T20:58:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T20:58:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T20:58:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T20:58:05.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:58:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T20:58:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T20:58:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:58:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:58:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T20:58:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:58:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:58:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T20:58:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:58:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T20:58:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: failover / failback loop' 2024-03-23T20:58:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T20:58:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T20:58:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 20:58:06 TEST: failover / failback loop 2024-03-23T20:58:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 20:58:06 TEST: failover / failback loop 2024-03-23T20:58:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 20 2024-03-23T20:58:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T20:58:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T20:58:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T20:58:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:58:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:58:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:58:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:58:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:09.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:58:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:07 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:07' 2024-03-23T20:58:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T20:58:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:58:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:11 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:12' 2024-03-23T20:58:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:58:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:58:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:11' 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:58:11' 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:11' 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:58:11cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:58:12' 2024-03-23T20:58:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:11' 2024-03-23T20:58:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:58:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:58:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:58:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:12 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:11' 2024-03-23T20:58:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:58:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:58:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:12' 2024-03-23T20:58:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:58:12' 2024-03-23T20:58:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:12' 2024-03-23T20:58:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:58:12cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:58:11' 2024-03-23T20:58:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:12' 2024-03-23T20:58:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:58:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:58:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T20:58:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:58:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T20:58:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:58:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:58:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:58:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:58:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:58:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:58:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:58:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T20:58:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:58:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:58:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:58:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T20:58:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T20:58:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T20:58:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:58:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:58:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:58:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T20:58:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:58:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:58:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:17.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:17.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:58:17.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:58:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:58:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=19, entry_tid=0' 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:58:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:58:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:58:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:58:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=19, entry_tid=0' 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=19, entry_tid=0' -a 'object_number=0, tag_tid=19, entry_tid=0' = 'object_number=0, tag_tid=19, entry_tid=0' 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:16 2024-03-23T20:58:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2024-03-23T20:58:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:17' 2024-03-23T20:58:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:58:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:58:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:58:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:16' 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:58:16' 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:16' 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 20:58:16cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}2024-03-23 20:58:17' 2024-03-23T20:58:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:16' 2024-03-23T20:58:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:58:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:58:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T20:58:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:17 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:16' 2024-03-23T20:58:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:58:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:58:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2024-03-23T20:58:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:17' 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 20:58:17' 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:17' 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}44564463mirror.0smithi2042024-03-23 20:58:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 20:58:16' 2024-03-23T20:58:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:17' 2024-03-23T20:58:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:58:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:58:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T20:58:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T20:58:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:58:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:58:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:58:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:21 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:22' 2024-03-23T20:58:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:58:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:58:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:21' 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:58:21' 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:21' 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:58:21cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:58:22' 2024-03-23T20:58:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:21' 2024-03-23T20:58:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:58:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:58:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:22 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:21' 2024-03-23T20:58:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:58:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:58:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:22' 2024-03-23T20:58:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:58:22' 2024-03-23T20:58:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:22' 2024-03-23T20:58:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:58:22cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:58:21' 2024-03-23T20:58:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:22' 2024-03-23T20:58:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:58:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:58:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T20:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T20:58:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:58:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:58:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:58:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:58:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:58:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:58:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:58:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:58:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:58:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:58:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:58:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:58:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:58:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:58:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:58:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:58:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:27.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=23, entry_tid=0' 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:58:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=23, entry_tid=0' 2024-03-23T20:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=23, entry_tid=0' -a 'object_number=0, tag_tid=23, entry_tid=0' = 'object_number=0, tag_tid=23, entry_tid=0' 2024-03-23T20:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T20:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:27 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:27' 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:58:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:58:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:58:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:27' 2024-03-23T20:58:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 20:58:27' 2024-03-23T20:58:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:27' 2024-03-23T20:58:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 20:58:27cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}2024-03-23 20:58:27' 2024-03-23T20:58:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:27' 2024-03-23T20:58:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:58:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:58:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:58:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:27 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:27' 2024-03-23T20:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:58:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:58:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2024-03-23T20:58:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:27' 2024-03-23T20:58:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:58:27' 2024-03-23T20:58:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:27' 2024-03-23T20:58:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}42894296mirror.0smithi2042024-03-23 20:58:27cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 20:58:27' 2024-03-23T20:58:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:27' 2024-03-23T20:58:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:58:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:58:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T20:58:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T20:58:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T20:58:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:58:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:58:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopping"' 2024-03-23T20:58:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopping"' 2024-03-23T20:58:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:58:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:58:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:31 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:32' 2024-03-23T20:58:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:58:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:58:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:31' 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:58:31' 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:31' 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:58:31cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:58:32' 2024-03-23T20:58:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:31' 2024-03-23T20:58:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:58:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:58:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:58:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:32 2024-03-23T20:58:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:31' 2024-03-23T20:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:58:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:32' 2024-03-23T20:58:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:58:32' 2024-03-23T20:58:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:32' 2024-03-23T20:58:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:58:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:58:31' 2024-03-23T20:58:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:32' 2024-03-23T20:58:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:58:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:58:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T20:58:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:58:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T20:58:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:58:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:58:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:58:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:58:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T20:58:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:58:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:58:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:58:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:58:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:58:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:58:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:58:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:58:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:58:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:58:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:58:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:58:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T20:58:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:58:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:58:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:58:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T20:58:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T20:58:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T20:58:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:58:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:58:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:58:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:58:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T20:58:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T20:58:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:58:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:58:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:58:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:58:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=24, entry_tid=0' 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:58:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:58:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:58:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:58:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=24, entry_tid=0' 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=24, entry_tid=0' -a 'object_number=0, tag_tid=24, entry_tid=0' = 'object_number=0, tag_tid=24, entry_tid=0' 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:58:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:41 2024-03-23T20:58:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2024-03-23T20:58:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:37' 2024-03-23T20:58:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:58:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:58:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:58:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:41' 2024-03-23T20:58:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:58:41' 2024-03-23T20:58:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:41' 2024-03-23T20:58:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 20:58:41cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}2024-03-23 20:58:37' 2024-03-23T20:58:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:41' 2024-03-23T20:58:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:58:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:58:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T20:58:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:37 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:41' 2024-03-23T20:58:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:58:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:58:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2024-03-23T20:58:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:37' 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 20:58:37' 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:37' 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}44564463mirror.0smithi2042024-03-23 20:58:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 20:58:41' 2024-03-23T20:58:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:37' 2024-03-23T20:58:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:58:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:58:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T20:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T20:58:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:58:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:58:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:58:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:58:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:58:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:58:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:58:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:58:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:58:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:58:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:58:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:58:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:58:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:58:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:58:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:46 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:47' 2024-03-23T20:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:58:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:58:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:46' 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:58:46' 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:46' 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:58:46cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:58:47' 2024-03-23T20:58:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:46' 2024-03-23T20:58:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:58:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:58:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:58:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:58:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:47 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:46' 2024-03-23T20:58:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:58:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:58:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:58:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:47' 2024-03-23T20:58:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:58:47' 2024-03-23T20:58:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:47' 2024-03-23T20:58:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:58:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:58:46' 2024-03-23T20:58:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:47' 2024-03-23T20:58:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:58:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:58:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:58:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T20:58:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:58:49.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T20:58:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:58:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:58:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:58:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:58:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:58:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:58:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:58:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:58:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:58:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:58:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:58:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:58:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:58:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:58:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:58:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:58:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:58:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:58:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:58:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:58:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:58:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=28, entry_tid=0' 2024-03-23T20:58:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:58:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:58:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:58:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:58:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:58:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:58:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:58:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:58:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:58:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:58:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:58:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:58:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:58:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=28, entry_tid=0' 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=28, entry_tid=0' -a 'object_number=0, tag_tid=28, entry_tid=0' = 'object_number=0, tag_tid=28, entry_tid=0' 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T20:58:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:58:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:52 2024-03-23T20:58:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:58:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2024-03-23T20:58:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:52' 2024-03-23T20:58:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:58:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:58:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:58:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:52' 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 20:58:52' 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:52' 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 20:58:52cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}2024-03-23 20:58:52' 2024-03-23T20:58:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:52' 2024-03-23T20:58:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:58:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:58:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:58:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:58:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:58:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:58:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:58:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:58:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2024-03-23T20:58:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:58:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:52 2024-03-23T20:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:52' 2024-03-23T20:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:58:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2024-03-23T20:58:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:58:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:58:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:58:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:58:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:58:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:52' 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:58:52' 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:58:52' 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:58:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:58:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}42894296mirror.0smithi2042024-03-23 20:58:52cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 20:58:52' 2024-03-23T20:58:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:58:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:58:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:58:52' 2024-03-23T20:58:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:58:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:58:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T20:58:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T20:58:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:58:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T20:58:56.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:58:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:58:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:58:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:58:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:58:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:58:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:58:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:58:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:58:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:58:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:58:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:58:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:58:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:58:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:58:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:58:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:58:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:58:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:58:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:58:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:58:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:58:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:58:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:58:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:58:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:58:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:58:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:58:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:58:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:58:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:58:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:58:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:58:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:58:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:58:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:58:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:58:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:58:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:59:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:59:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:57 2024-03-23T20:59:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:58:57' 2024-03-23T20:59:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T20:59:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:01 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:02' 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:59:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:59:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:01' 2024-03-23T20:59:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:59:01' 2024-03-23T20:59:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:01' 2024-03-23T20:59:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:59:01cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:59:02' 2024-03-23T20:59:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:01' 2024-03-23T20:59:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:59:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:59:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:59:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:59:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:02 2024-03-23T20:59:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:01' 2024-03-23T20:59:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:59:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:59:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:03.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:02' 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:59:02' 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:02' 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:59:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:59:01' 2024-03-23T20:59:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:02' 2024-03-23T20:59:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:59:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:59:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T20:59:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:59:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T20:59:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:59:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:59:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:59:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:59:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T20:59:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:59:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:59:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:59:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:59:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:59:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:59:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:59:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:59:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:59:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:59:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T20:59:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:59:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T20:59:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:59:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:59:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:59:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T20:59:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:59:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:59:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:59:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:59:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T20:59:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:59:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T20:59:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T20:59:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:59:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:59:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:59:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:59:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:59:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:59:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:59:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:59:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:59:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:59:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=29, entry_tid=0' 2024-03-23T20:59:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T20:59:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T20:59:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:59:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:59:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:59:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:59:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:59:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:59:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:59:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:59:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:59:11.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:59:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=29, entry_tid=0' 2024-03-23T20:59:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=29, entry_tid=0' -a 'object_number=0, tag_tid=29, entry_tid=0' = 'object_number=0, tag_tid=29, entry_tid=0' 2024-03-23T20:59:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:59:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:59:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:11 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:07' 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:59:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:59:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:59:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:59:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:59:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:11' 2024-03-23T20:59:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:59:11' 2024-03-23T20:59:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:11' 2024-03-23T20:59:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 20:59:11cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}2024-03-23 20:59:07' 2024-03-23T20:59:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:11' 2024-03-23T20:59:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:59:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:59:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:59:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T20:59:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:59:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2024-03-23T20:59:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:07 2024-03-23T20:59:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:59:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:11' 2024-03-23T20:59:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:59:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:59:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2024-03-23T20:59:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:59:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:59:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:07' 2024-03-23T20:59:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 20:59:07' 2024-03-23T20:59:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:07' 2024-03-23T20:59:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}44564463mirror.0smithi2042024-03-23 20:59:07cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 20:59:11' 2024-03-23T20:59:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:07' 2024-03-23T20:59:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:59:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:59:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:59:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:59:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T20:59:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T20:59:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:59:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T20:59:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T20:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:59:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:59:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:59:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:59:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:59:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:59:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:59:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:59:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:59:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:59:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:16 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:17' 2024-03-23T20:59:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:59:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:59:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:16' 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:59:16' 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:16' 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:59:16cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:59:17' 2024-03-23T20:59:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:16' 2024-03-23T20:59:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:59:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:59:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:59:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:59:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:17 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:16' 2024-03-23T20:59:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:59:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:59:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:17' 2024-03-23T20:59:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:59:17' 2024-03-23T20:59:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:17' 2024-03-23T20:59:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:59:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:59:16' 2024-03-23T20:59:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:17' 2024-03-23T20:59:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:59:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:59:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T20:59:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:59:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T20:59:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:59:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:59:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:59:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:59:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:59:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:59:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:59:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:59:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:59:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:59:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:59:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:59:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:59:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:59:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:59:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:59:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:59:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:59:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:59:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:59:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:59:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:59:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:59:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:59:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:59:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:59:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:59:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:59:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:59:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:59:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:59:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:59:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:59:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:59:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:59:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:59:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:59:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:59:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:59:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=33, entry_tid=0' 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:59:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:59:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:59:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:59:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:59:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:59:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:59:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:59:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=33, entry_tid=0' 2024-03-23T20:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=33, entry_tid=0' -a 'object_number=0, tag_tid=33, entry_tid=0' = 'object_number=0, tag_tid=33, entry_tid=0' 2024-03-23T20:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T20:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:22 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:59:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2024-03-23T20:59:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:22' 2024-03-23T20:59:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:59:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:59:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:59:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:59:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:59:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:22' 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 20:59:22' 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:22' 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 20:59:22cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}2024-03-23 20:59:22' 2024-03-23T20:59:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:22' 2024-03-23T20:59:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:59:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:59:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:59:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:59:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:59:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:59:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:59:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2024-03-23T20:59:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:22 2024-03-23T20:59:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:59:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:22' 2024-03-23T20:59:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:59:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:59:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2024-03-23T20:59:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:59:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:59:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:22' 2024-03-23T20:59:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:59:22' 2024-03-23T20:59:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:22' 2024-03-23T20:59:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}42894296mirror.0smithi2042024-03-23 20:59:22cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 20:59:22' 2024-03-23T20:59:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:22' 2024-03-23T20:59:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:59:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:59:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:59:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:59:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T20:59:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T20:59:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T20:59:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:59:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:59:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:59:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:59:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:59:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:59:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:59:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:59:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:59:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:59:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:59:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:59:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:59:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:59:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:59:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:59:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:59:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:27 2024-03-23T20:59:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:27' 2024-03-23T20:59:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T20:59:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:59:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:31 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:32' 2024-03-23T20:59:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:59:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:59:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:31' 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:59:31' 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:31' 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:59:31cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 20:59:32' 2024-03-23T20:59:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:31' 2024-03-23T20:59:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:59:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:59:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:59:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:32 2024-03-23T20:59:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:31' 2024-03-23T20:59:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:59:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:59:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:32' 2024-03-23T20:59:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:59:32' 2024-03-23T20:59:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:32' 2024-03-23T20:59:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:59:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:59:31' 2024-03-23T20:59:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:32' 2024-03-23T20:59:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:59:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:59:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T20:59:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:59:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T20:59:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:59:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:59:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:35.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:59:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T20:59:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:59:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:59:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:59:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:59:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:59:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:59:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T20:59:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:59:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T20:59:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:59:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:59:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:59:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:59:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:59:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:59:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:59:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:59:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:59:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:59:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:59:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=34, entry_tid=0' 2024-03-23T20:59:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T20:59:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:59:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:59:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T20:59:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:59:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:59:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:59:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:59:37.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T20:59:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=34, entry_tid=0' 2024-03-23T20:59:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=34, entry_tid=0' -a 'object_number=0, tag_tid=34, entry_tid=0' = 'object_number=0, tag_tid=34, entry_tid=0' 2024-03-23T20:59:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T20:59:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T20:59:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:36 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:37' 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:59:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:59:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:59:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:59:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:59:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:36' 2024-03-23T20:59:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 20:59:36' 2024-03-23T20:59:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:36' 2024-03-23T20:59:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 20:59:36cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}2024-03-23 20:59:37' 2024-03-23T20:59:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:36' 2024-03-23T20:59:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:59:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:59:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:59:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:59:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T20:59:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:37 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:36' 2024-03-23T20:59:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:59:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:59:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2024-03-23T20:59:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:59:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:59:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:37' 2024-03-23T20:59:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 20:59:37' 2024-03-23T20:59:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:37' 2024-03-23T20:59:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}44564463mirror.0smithi2042024-03-23 20:59:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 20:59:36' 2024-03-23T20:59:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:37' 2024-03-23T20:59:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:59:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:59:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:59:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:59:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T20:59:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T20:59:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:59:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T20:59:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T20:59:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:59:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:59:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:59:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:59:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:59:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:59:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:59:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:59:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T20:59:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T20:59:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:59:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T20:59:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T20:59:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T20:59:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T20:59:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T20:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T20:59:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T20:59:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T20:59:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T20:59:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T20:59:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T20:59:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:41 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:59:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:42' 2024-03-23T20:59:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:59:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:59:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:41' 2024-03-23T20:59:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 20:59:41' 2024-03-23T20:59:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:41' 2024-03-23T20:59:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 20:59:41cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedstopped2024-03-23 20:59:42' 2024-03-23T20:59:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:41' 2024-03-23T20:59:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:59:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:59:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T20:59:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:42 2024-03-23T20:59:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:41' 2024-03-23T20:59:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T20:59:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T20:59:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:47 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T20:59:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T20:59:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:46' 2024-03-23T20:59:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T20:59:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T20:59:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T20:59:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:47' 2024-03-23T20:59:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 20:59:47' 2024-03-23T20:59:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T20:59:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:47' 2024-03-23T20:59:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 20:59:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 20:59:46' 2024-03-23T20:59:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:47' 2024-03-23T20:59:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T20:59:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T20:59:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T20:59:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T20:59:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T20:59:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T20:59:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T20:59:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T20:59:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T20:59:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:59:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:59:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:59:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:59:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:59:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:59:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:59:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T20:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:59:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:59:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:59:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T20:59:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T20:59:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T20:59:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T20:59:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T20:59:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T20:59:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:59:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:59:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:59:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:59:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T20:59:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T20:59:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T20:59:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T20:59:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T20:59:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T20:59:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T20:59:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T20:59:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T20:59:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T20:59:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T20:59:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T20:59:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T20:59:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T20:59:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T20:59:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T20:59:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T20:59:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T20:59:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:59:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:59:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:59:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:59:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:59:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:59:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:59:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:59:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:59:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:56.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=38, entry_tid=0' 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T20:59:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T20:59:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T20:59:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T20:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T20:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T20:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T20:59:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T20:59:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=38, entry_tid=0' 2024-03-23T20:59:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=38, entry_tid=0' -a 'object_number=0, tag_tid=38, entry_tid=0' = 'object_number=0, tag_tid=38, entry_tid=0' 2024-03-23T20:59:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T20:59:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T20:59:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T20:59:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:59:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:57 2024-03-23T20:59:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T20:59:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:59:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2024-03-23T20:59:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:56' 2024-03-23T20:59:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T20:59:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T20:59:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T20:59:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:59:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:59:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:57' 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 20:59:57' 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:57' 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 20:59:57cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}2024-03-23 20:59:56' 2024-03-23T20:59:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:57' 2024-03-23T20:59:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T20:59:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T20:59:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T20:59:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T20:59:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T20:59:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T20:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T20:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T20:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T20:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2024-03-23T20:59:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T20:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:56 2024-03-23T20:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T20:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T20:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T20:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T20:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 20:59:57' 2024-03-23T20:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T20:59:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T20:59:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T20:59:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2024-03-23T20:59:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T20:59:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:59:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T20:59:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:59:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T20:59:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T20:59:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T20:59:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T20:59:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:56' 2024-03-23T20:59:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 20:59:56' 2024-03-23T20:59:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T20:59:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 20:59:56' 2024-03-23T20:59:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T20:59:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T20:59:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T20:59:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}42894296mirror.0smithi2042024-03-23 20:59:56cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 20:59:57' 2024-03-23T20:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T20:59:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T20:59:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 20:59:56' 2024-03-23T20:59:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T20:59:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T20:59:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T20:59:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T20:59:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T20:59:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T20:59:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T20:59:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T20:59:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T20:59:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T20:59:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T20:59:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T20:59:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T20:59:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T20:59:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T20:59:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T20:59:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T20:59:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T20:59:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T20:59:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T20:59:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T20:59:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T20:59:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T20:59:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T20:59:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T20:59:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T20:59:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T20:59:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T20:59:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T20:59:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T20:59:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:00:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:00:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:02.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:02.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:02.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:02.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:00:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:00:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:01 2024-03-23T21:00:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:02' 2024-03-23T21:00:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:01' 2024-03-23T21:00:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:00:01' 2024-03-23T21:00:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:01' 2024-03-23T21:00:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:00:01cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:00:02' 2024-03-23T21:00:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:01' 2024-03-23T21:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:00:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:02 2024-03-23T21:00:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:01' 2024-03-23T21:00:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:02' 2024-03-23T21:00:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:00:02' 2024-03-23T21:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:02' 2024-03-23T21:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:00:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:00:01' 2024-03-23T21:00:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:02' 2024-03-23T21:00:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:00:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:00:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:00:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:00:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:00:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:00:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:00:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:05.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:00:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:00:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:00:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:00:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:00:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:00:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:00:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:00:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:00:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:00:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:00:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:00:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:00:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:00:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:00:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:00:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:00:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:00:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:00:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:00:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:00:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:00:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:00:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:00:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:00:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:00:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:00:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:00:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:00:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:00:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=39, entry_tid=0' 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:00:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:00:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:00:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:00:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=39, entry_tid=0' 2024-03-23T21:00:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=39, entry_tid=0' -a 'object_number=0, tag_tid=39, entry_tid=0' = 'object_number=0, tag_tid=39, entry_tid=0' 2024-03-23T21:00:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:00:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:00:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:06 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:07' 2024-03-23T21:00:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:00:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:00:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:00:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:00:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:00:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:06' 2024-03-23T21:00:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:00:06' 2024-03-23T21:00:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:06' 2024-03-23T21:00:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:00:06cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}2024-03-23 21:00:07' 2024-03-23T21:00:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:06' 2024-03-23T21:00:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:00:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:00:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:00:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:00:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:00:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2024-03-23T21:00:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:07 2024-03-23T21:00:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:00:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:06' 2024-03-23T21:00:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:00:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:00:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2024-03-23T21:00:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:00:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:00:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:07' 2024-03-23T21:00:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:00:07' 2024-03-23T21:00:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:07' 2024-03-23T21:00:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}44564463mirror.0smithi2042024-03-23 21:00:07cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:00:06' 2024-03-23T21:00:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:07' 2024-03-23T21:00:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:00:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:00:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:00:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:00:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:00:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:00:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:00:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:00:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:00:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:00:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:00:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:00:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:00:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:00:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:00:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:00:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:00:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:00:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:11 2024-03-23T21:00:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:00:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:12' 2024-03-23T21:00:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:11' 2024-03-23T21:00:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:00:11' 2024-03-23T21:00:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:11' 2024-03-23T21:00:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:00:11cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedstopped2024-03-23 21:00:12' 2024-03-23T21:00:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:11' 2024-03-23T21:00:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:12 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:11' 2024-03-23T21:00:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:00:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:00:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:17 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:16' 2024-03-23T21:00:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:17' 2024-03-23T21:00:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:00:17' 2024-03-23T21:00:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:17' 2024-03-23T21:00:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:00:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:00:16' 2024-03-23T21:00:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:17' 2024-03-23T21:00:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:00:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:00:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:00:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:00:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:19.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:00:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:00:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:00:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:00:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:00:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:00:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:00:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:00:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:00:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:00:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:00:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:00:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:00:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:00:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:00:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:00:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:00:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:00:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:00:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=43, entry_tid=0' 2024-03-23T21:00:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:00:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:00:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=43, entry_tid=0' 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=43, entry_tid=0' -a 'object_number=0, tag_tid=43, entry_tid=0' = 'object_number=0, tag_tid=43, entry_tid=0' 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:00:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:27 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:26' 2024-03-23T21:00:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:00:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:00:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:00:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:00:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:00:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:27' 2024-03-23T21:00:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:00:27' 2024-03-23T21:00:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:27' 2024-03-23T21:00:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:00:27cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}2024-03-23 21:00:26' 2024-03-23T21:00:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:27' 2024-03-23T21:00:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:00:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:00:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:00:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:00:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:00:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2024-03-23T21:00:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:26 2024-03-23T21:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:27' 2024-03-23T21:00:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:00:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:00:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2024-03-23T21:00:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:00:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:00:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:26' 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:00:26' 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:26' 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}42894296mirror.0smithi2042024-03-23 21:00:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:00:27' 2024-03-23T21:00:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:26' 2024-03-23T21:00:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:00:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:00:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:00:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:00:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:00:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:00:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:00:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:00:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:00:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:00:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:00:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:00:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:32.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:31 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:32' 2024-03-23T21:00:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:31' 2024-03-23T21:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:00:31' 2024-03-23T21:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:31' 2024-03-23T21:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:00:31cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:00:32' 2024-03-23T21:00:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:31' 2024-03-23T21:00:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:00:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:32 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:31' 2024-03-23T21:00:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:32' 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:00:32' 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:32' 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:00:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:00:31' 2024-03-23T21:00:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:32' 2024-03-23T21:00:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:00:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:00:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:00:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:00:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:00:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:00:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:00:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:00:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:00:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:00:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:00:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:00:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:00:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:00:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:00:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:00:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:00:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:00:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:00:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:00:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:00:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:00:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:00:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:00:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:00:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:00:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:00:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:00:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:00:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:00:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:00:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:00:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:00:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=44, entry_tid=0' 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:00:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:00:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:00:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:00:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=44, entry_tid=0' 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=44, entry_tid=0' -a 'object_number=0, tag_tid=44, entry_tid=0' = 'object_number=0, tag_tid=44, entry_tid=0' 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:00:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:36 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:37' 2024-03-23T21:00:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:00:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:00:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:00:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:00:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:00:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:36' 2024-03-23T21:00:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:00:36' 2024-03-23T21:00:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:36' 2024-03-23T21:00:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:00:36cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}2024-03-23 21:00:37' 2024-03-23T21:00:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:36' 2024-03-23T21:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:00:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:00:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:00:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:00:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:00:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2024-03-23T21:00:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:37 2024-03-23T21:00:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:00:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:36' 2024-03-23T21:00:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:00:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:00:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2024-03-23T21:00:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:00:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:00:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:37' 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:00:37' 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:37' 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}44564463mirror.0smithi2042024-03-23 21:00:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:00:36' 2024-03-23T21:00:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:37' 2024-03-23T21:00:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:00:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:00:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:00:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:00:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:00:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:00:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:00:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:00:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:00:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:00:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:00:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:00:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:00:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:00:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:42.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:00:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:00:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:00:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:00:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:00:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:00:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:00:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:00:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:00:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:00:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:00:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:00:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:00:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:00:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:41 2024-03-23T21:00:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:00:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:42' 2024-03-23T21:00:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:41' 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:00:41' 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:41' 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:00:41cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedstopped2024-03-23 21:00:42' 2024-03-23T21:00:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:41' 2024-03-23T21:00:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:00:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:42 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:41' 2024-03-23T21:00:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:00:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:00:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:47 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:46' 2024-03-23T21:00:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:47' 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:00:47' 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:47' 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:00:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:00:46' 2024-03-23T21:00:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:47' 2024-03-23T21:00:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:00:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:00:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:00:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:00:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:00:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:00:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:00:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:00:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:00:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:00:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:00:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:00:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:00:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:00:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:00:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:00:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:00:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:00:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:00:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:00:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:00:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:00:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=45, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=48, entry_tid=0' 2024-03-23T21:00:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:00:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:00:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:00:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:00:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=45, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=45, entry_tid=0' 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=48, entry_tid=0' -a 'object_number=0, tag_tid=48, entry_tid=0' = 'object_number=0, tag_tid=45, entry_tid=0' 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=45, entry_tid=0' '!=' '' 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=45, entry_tid=0' = '' 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=45, entry_tid=0' 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=48, entry_tid=0' 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:00:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=48 2024-03-23T21:00:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=45, entry_tid=0' 2024-03-23T21:00:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:00:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:00:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=45 2024-03-23T21:00:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=48, entry_tid=0' 2024-03-23T21:00:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:00:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:00:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-03-23T21:00:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=45, entry_tid=0' 2024-03-23T21:00:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:00:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:00:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-03-23T21:00:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 48 = 45 -a 0 -le 0 2024-03-23T21:00:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:00:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:00:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:00:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:00:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:00:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:00:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:00:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:00:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:00:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:00:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:00:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:00:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:00:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:00:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=48, entry_tid=0' 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:00:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:00:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:00:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:00:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:00:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:00:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:00:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:00:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:00:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:00:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=48, entry_tid=0' 2024-03-23T21:00:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=48, entry_tid=0' -a 'object_number=0, tag_tid=48, entry_tid=0' = 'object_number=0, tag_tid=48, entry_tid=0' 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:00:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:52 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:00:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2024-03-23T21:00:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:52' 2024-03-23T21:00:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:00:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:00:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:00:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:00:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:00:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:52' 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:00:52' 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:52' 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:00:52cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}2024-03-23 21:00:52' 2024-03-23T21:00:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:52' 2024-03-23T21:00:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:00:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:00:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:00:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:00:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:00:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:00:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:00:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2024-03-23T21:00:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:52 2024-03-23T21:00:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:00:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:00:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:52' 2024-03-23T21:00:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:00:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:00:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2024-03-23T21:00:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:00:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:00:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:52' 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:00:52' 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:52' 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}42894296mirror.0smithi2042024-03-23 21:00:52cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:00:52' 2024-03-23T21:00:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:52' 2024-03-23T21:00:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:00:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:00:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:00:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:00:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:00:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:00:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:00:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:00:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:00:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:00:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:00:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:00:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:00:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:00:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:00:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:00:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:00:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:00:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:00:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:00:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:00:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:00:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:00:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:00:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:00:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:00:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:00:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:00:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:00:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:00:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:00:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:00:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:00:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:00:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:00:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:00:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:00:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:00:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:00:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:00:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:56 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:57' 2024-03-23T21:00:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:00:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:00:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:00:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:00:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:00:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:00:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:00:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:00:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:00:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:56' 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:00:56' 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:56' 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:00:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:00:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:00:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:00:56cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:00:57' 2024-03-23T21:00:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:00:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:00:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:56' 2024-03-23T21:00:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:00:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:00:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:00:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:00:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:00:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:00:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:01:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:57 2024-03-23T21:01:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:00:56' 2024-03-23T21:01:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:57' 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:00:57' 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:00:57' 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:00:57cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:00:56' 2024-03-23T21:01:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:00:57' 2024-03-23T21:01:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:01.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:01:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:01:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:01:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:01:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:01:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:01:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:01:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:01:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:01:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:01:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:01:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:01:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:04.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:01:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:01:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:01:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:01:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:01:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:01:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:01:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:01:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:01:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:01:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:01:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:01:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:01:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:01:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:01:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:01:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=49, entry_tid=0' 2024-03-23T21:01:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:01:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=49, entry_tid=0' 2024-03-23T21:01:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=49, entry_tid=0' -a 'object_number=0, tag_tid=49, entry_tid=0' = 'object_number=0, tag_tid=49, entry_tid=0' 2024-03-23T21:01:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:01:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:01:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:01 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:02' 2024-03-23T21:01:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:01:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:01:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:01:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:01' 2024-03-23T21:01:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:01:01' 2024-03-23T21:01:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:01' 2024-03-23T21:01:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:01:01cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}2024-03-23 21:01:02' 2024-03-23T21:01:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:01' 2024-03-23T21:01:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:01:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:01:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:01:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:02 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:06' 2024-03-23T21:01:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:01:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:01:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2024-03-23T21:01:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:02' 2024-03-23T21:01:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:01:02' 2024-03-23T21:01:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:02' 2024-03-23T21:01:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}44564463mirror.0smithi2042024-03-23 21:01:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:01:06' 2024-03-23T21:01:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:02' 2024-03-23T21:01:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:01:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:01:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:01:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:01:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:01:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:01:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:01:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:01:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:01:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:01:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:06 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:08' 2024-03-23T21:01:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:01:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:01:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:11 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:12' 2024-03-23T21:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:11' 2024-03-23T21:01:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:01:11' 2024-03-23T21:01:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:11' 2024-03-23T21:01:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:01:11cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:01:12' 2024-03-23T21:01:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:11' 2024-03-23T21:01:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:01:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:12 2024-03-23T21:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:11' 2024-03-23T21:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:12' 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:01:12' 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:12' 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:01:12cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:01:11' 2024-03-23T21:01:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:12' 2024-03-23T21:01:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:01:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:01:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:01:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:01:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:01:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:01:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:01:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:01:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:01:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:01:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:01:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:01:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:01:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:01:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:01:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:01:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:01:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:01:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:01:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:01:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:01:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:01:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:01:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:01:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:01:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:01:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:01:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:01:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:01:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:01:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:01:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:01:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:01:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:01:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:01:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:01:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:01:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:01:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:01:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:01:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=53, entry_tid=0' 2024-03-23T21:01:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:01:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:01:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:01:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:01:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=53, entry_tid=0' 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=53, entry_tid=0' -a 'object_number=0, tag_tid=53, entry_tid=0' = 'object_number=0, tag_tid=53, entry_tid=0' 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:01:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:17 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2024-03-23T21:01:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:17' 2024-03-23T21:01:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:01:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:01:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:01:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:17' 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:01:17' 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:17' 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:01:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}2024-03-23 21:01:17' 2024-03-23T21:01:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:17' 2024-03-23T21:01:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:01:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:01:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:01:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:17 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:17' 2024-03-23T21:01:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:01:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:01:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2024-03-23T21:01:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:17' 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:01:17' 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:17' 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}42894296mirror.0smithi2042024-03-23 21:01:17cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:01:17' 2024-03-23T21:01:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:17' 2024-03-23T21:01:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:01:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:01:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:01:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:01:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:01:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:01:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:01:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:01:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:01:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:01:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:01:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:01:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:01:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:01:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:01:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:22.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:01:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:01:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:01:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:01:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:01:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:01:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:01:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:01:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:01:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:01:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:01:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:01:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:01:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:01:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:22 2024-03-23T21:01:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:22' 2024-03-23T21:01:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:01:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:01:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:26 2024-03-23T21:01:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:27' 2024-03-23T21:01:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:26' 2024-03-23T21:01:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:01:26' 2024-03-23T21:01:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:26' 2024-03-23T21:01:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:01:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:01:27' 2024-03-23T21:01:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:26' 2024-03-23T21:01:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:01:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:27 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:26' 2024-03-23T21:01:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:27' 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:01:27' 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:27' 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:01:27cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:01:26' 2024-03-23T21:01:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:27' 2024-03-23T21:01:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:01:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:01:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:01:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:01:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:01:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:01:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:01:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:01:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:01:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:01:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:01:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:01:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:01:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:01:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:01:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:01:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:01:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:01:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:01:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:01:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:01:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:01:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:01:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:01:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:01:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:01:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:01:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:01:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:01:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:01:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:01:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=54, entry_tid=0' 2024-03-23T21:01:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:01:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:01:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=54, entry_tid=0' 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=54, entry_tid=0' -a 'object_number=0, tag_tid=54, entry_tid=0' = 'object_number=0, tag_tid=54, entry_tid=0' 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:31 2024-03-23T21:01:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2024-03-23T21:01:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:32' 2024-03-23T21:01:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:01:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:01:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:01:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:31' 2024-03-23T21:01:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:01:31' 2024-03-23T21:01:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:31' 2024-03-23T21:01:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:01:31cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}2024-03-23 21:01:32' 2024-03-23T21:01:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:31' 2024-03-23T21:01:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:01:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:01:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:01:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2024-03-23T21:01:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:32 2024-03-23T21:01:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:31' 2024-03-23T21:01:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:01:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:01:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2024-03-23T21:01:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:32' 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:01:32' 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:32' 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}44564463mirror.0smithi2042024-03-23 21:01:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:01:31' 2024-03-23T21:01:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:32' 2024-03-23T21:01:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:01:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:01:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:01:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:01:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:01:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:01:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:01:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:01:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:01:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:01:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:01:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:01:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:01:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:01:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:01:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:01:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:36 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:37' 2024-03-23T21:01:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:36' 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:01:36' 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:36' 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:01:36cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:01:37' 2024-03-23T21:01:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:36' 2024-03-23T21:01:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:01:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:37 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:36' 2024-03-23T21:01:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:37' 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:01:37' 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:37' 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:01:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:01:36' 2024-03-23T21:01:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:37' 2024-03-23T21:01:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:01:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:01:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:01:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:01:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:01:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:01:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:01:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:01:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:01:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:01:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:01:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:01:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:01:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:01:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:01:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:01:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:01:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:01:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:01:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:01:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:01:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:01:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:01:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:01:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=58, entry_tid=0' 2024-03-23T21:01:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:01:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:01:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:01:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:01:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:01:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:01:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=58, entry_tid=0' 2024-03-23T21:01:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=58, entry_tid=0' -a 'object_number=0, tag_tid=58, entry_tid=0' = 'object_number=0, tag_tid=58, entry_tid=0' 2024-03-23T21:01:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:01:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:42 2024-03-23T21:01:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2024-03-23T21:01:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:42' 2024-03-23T21:01:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:01:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:01:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:01:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:42' 2024-03-23T21:01:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:01:42' 2024-03-23T21:01:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:42' 2024-03-23T21:01:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:01:42cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}2024-03-23 21:01:42' 2024-03-23T21:01:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:42' 2024-03-23T21:01:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:01:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:01:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:01:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:42 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:42' 2024-03-23T21:01:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:01:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:01:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2024-03-23T21:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:42' 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:01:42' 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:42' 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}42894296mirror.0smithi2042024-03-23 21:01:42cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:01:42' 2024-03-23T21:01:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:42' 2024-03-23T21:01:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:01:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:01:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:01:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:01:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:01:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:01:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:01:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:01:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:01:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:01:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:01:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:01:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:01:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:01:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:01:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:01:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:01:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:01:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:01:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:01:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:01:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:01:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:01:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:01:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:01:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:01:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:01:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:01:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:01:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:01:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:46 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:47' 2024-03-23T21:01:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:46' 2024-03-23T21:01:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:01:46' 2024-03-23T21:01:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:46' 2024-03-23T21:01:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:01:46cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:01:47' 2024-03-23T21:01:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:49.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:46' 2024-03-23T21:01:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:01:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:01:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:47 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:46' 2024-03-23T21:01:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:01:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:01:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:01:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:47' 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:01:47' 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:47' 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:01:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:01:46' 2024-03-23T21:01:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:47' 2024-03-23T21:01:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:01:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:01:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:01:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:01:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:01:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:01:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:01:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:01:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:01:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:01:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:01:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:01:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Starting"' 2024-03-23T21:01:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Starting"' 2024-03-23T21:01:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:01:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:01:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:01:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:01:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:01:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:01:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:01:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:01:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:01:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:01:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:01:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:01:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:01:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:01:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:01:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:01:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:01:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:01:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:01:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:01:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:01:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:01:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:01:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=59, entry_tid=0' 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:01:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:01:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:01:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:01:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:01:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:01:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:01:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:01:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:01:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=59, entry_tid=0' 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=59, entry_tid=0' -a 'object_number=0, tag_tid=59, entry_tid=0' = 'object_number=0, tag_tid=59, entry_tid=0' 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:01:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:51 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2024-03-23T21:01:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:52' 2024-03-23T21:01:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:01:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:01:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:01:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:51' 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:01:51' 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:51' 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:01:51cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}2024-03-23 21:01:52' 2024-03-23T21:01:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:51' 2024-03-23T21:01:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:01:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:01:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:01:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:01:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:01:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:01:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:01:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:01:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:01:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2024-03-23T21:01:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:01:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:52 2024-03-23T21:01:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:01:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:01:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:01:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:01:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:56' 2024-03-23T21:01:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:01:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:01:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:01:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2024-03-23T21:01:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:01:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:01:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:01:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:01:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:01:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:01:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:52' 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:01:52' 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:01:52' 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:01:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:01:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":12.0,"entries_behind_primary":0,"entries_per_second":0.5,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}44564463mirror.0smithi2042024-03-23 21:01:57cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:01:56' 2024-03-23T21:01:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:01:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:01:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:01:57' 2024-03-23T21:01:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:01:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:01:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:01:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:01:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:01:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:01:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:01:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:01:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:01:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:01:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:01:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:01:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:01:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:01:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:01:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:01:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:01:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:01:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:01:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:01:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:01:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:01:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:01:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:01:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:01:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:01:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:01:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:01:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:01:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:01:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:01:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:01:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:01:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:01:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:01:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:01:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:01:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:01:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:01:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:01:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:01:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:01:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:01:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:01:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:01:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:01:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:01:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:02:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:02:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:02:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:02:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:02:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:02:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:02:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:02:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:02:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:02:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:02:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:02:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:56 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:01:58' 2024-03-23T21:02:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:02:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:01 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:02' 2024-03-23T21:02:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:02:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:02:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:01' 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:02:01' 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:01' 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:02:01cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:02:02' 2024-03-23T21:02:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:01' 2024-03-23T21:02:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:02:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:02:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:02:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:02 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:01' 2024-03-23T21:02:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:02:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:02:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:02' 2024-03-23T21:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:02:02' 2024-03-23T21:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:02' 2024-03-23T21:02:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:02:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:02:01' 2024-03-23T21:02:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:02' 2024-03-23T21:02:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:02:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:02:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:02:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:02:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:02:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:02:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:02:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:02:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:02:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:02:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:02:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:02:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:02:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:02:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:02:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:02:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:02:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:02:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:02:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:02:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:02:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:08.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:02:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:02:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:02:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:02:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:02:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:02:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:02:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:02:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:02:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:02:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:02:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:02:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:02:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:02:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:02:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:02:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:02:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:02:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:02:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=63, entry_tid=0' 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:02:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=63, entry_tid=0' 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=63, entry_tid=0' -a 'object_number=0, tag_tid=63, entry_tid=0' = 'object_number=0, tag_tid=63, entry_tid=0' 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:02:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:07 2024-03-23T21:02:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:02:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2024-03-23T21:02:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:07' 2024-03-23T21:02:10.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:02:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:02:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:02:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:02:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:02:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:07' 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:02:07' 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:07' 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:02:07cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}2024-03-23 21:02:07' 2024-03-23T21:02:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:07' 2024-03-23T21:02:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:02:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:02:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:02:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2024-03-23T21:02:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:07 2024-03-23T21:02:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:07' 2024-03-23T21:02:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:02:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:02:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2024-03-23T21:02:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:02:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:02:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:07' 2024-03-23T21:02:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:02:07' 2024-03-23T21:02:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:07' 2024-03-23T21:02:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}42894296mirror.0smithi2042024-03-23 21:02:07cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:02:07' 2024-03-23T21:02:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:07' 2024-03-23T21:02:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:02:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:02:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:02:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:02:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:02:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:02:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:02:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:02:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:02:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:02:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:02:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:02:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:02:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:02:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:02:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:02:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:02:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:02:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:02:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:02:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:02:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:02:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:12 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:12' 2024-03-23T21:02:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:02:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:02:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:12 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:12' 2024-03-23T21:02:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:02:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:16 2024-03-23T21:02:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:17' 2024-03-23T21:02:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:02:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:20.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:02:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:16' 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:02:16' 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:16' 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:02:16cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:02:17' 2024-03-23T21:02:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:16' 2024-03-23T21:02:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:02:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:02:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:02:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:02:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:17 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:16' 2024-03-23T21:02:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:02:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:02:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:21.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:21.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:21.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:17' 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:02:17' 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:17' 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:02:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:02:16' 2024-03-23T21:02:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:17' 2024-03-23T21:02:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:02:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:02:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:02:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:02:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:02:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:02:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:02:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:02:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:02:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:02:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:02:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:02:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:02:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:02:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:02:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:02:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:02:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:02:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:02:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:02:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:02:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:02:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:02:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:02:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:02:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:02:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:02:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:02:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:02:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:02:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:02:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:02:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:02:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=64, entry_tid=0' 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:02:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:02:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:02:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:02:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=64, entry_tid=0' 2024-03-23T21:02:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=64, entry_tid=0' -a 'object_number=0, tag_tid=64, entry_tid=0' = 'object_number=0, tag_tid=64, entry_tid=0' 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:02:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:21 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:22' 2024-03-23T21:02:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:02:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:02:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:02:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:02:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:02:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:21' 2024-03-23T21:02:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:02:21' 2024-03-23T21:02:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:21' 2024-03-23T21:02:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:02:21cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}2024-03-23 21:02:22' 2024-03-23T21:02:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:21' 2024-03-23T21:02:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:02:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:02:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:02:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:02:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:02:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:22 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:21' 2024-03-23T21:02:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:02:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:02:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2024-03-23T21:02:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:02:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:02:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:22' 2024-03-23T21:02:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:02:22' 2024-03-23T21:02:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:22' 2024-03-23T21:02:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}44564463mirror.0smithi2042024-03-23 21:02:22cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:02:21' 2024-03-23T21:02:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:22' 2024-03-23T21:02:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:02:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:02:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:02:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:02:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:02:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:02:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:02:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:02:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:02:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:02:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:02:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:02:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:02:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:02:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:02:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:02:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:02:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:02:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:02:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:02:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:02:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:02:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:02:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:26 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:28' 2024-03-23T21:02:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:02:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:02:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:26' 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:02:26' 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:26' 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:02:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedstopped2024-03-23 21:02:28' 2024-03-23T21:02:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:26' 2024-03-23T21:02:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:02:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:02:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:02:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:28 2024-03-23T21:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:26' 2024-03-23T21:02:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:02:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:32 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:31' 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:02:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:02:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:32' 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:02:32' 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:32' 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:02:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:02:31' 2024-03-23T21:02:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:32' 2024-03-23T21:02:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:02:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:02:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:02:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:02:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:02:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:02:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:02:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:02:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:02:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:02:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:02:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:02:34.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:34.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:02:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:02:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:02:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:02:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:02:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:02:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:02:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:02:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:02:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:02:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:02:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:02:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:02:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:02:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:02:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:02:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:02:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:02:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:02:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:02:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:02:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:02:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:02:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:02:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:02:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:02:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:02:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:02:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:02:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:02:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:02:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:02:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:02:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:02:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:02:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:02:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:02:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:02:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=65, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=68, entry_tid=0' 2024-03-23T21:02:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:02:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:02:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:02:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:02:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=65, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=65, entry_tid=0' 2024-03-23T21:02:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=68, entry_tid=0' -a 'object_number=0, tag_tid=68, entry_tid=0' = 'object_number=0, tag_tid=65, entry_tid=0' 2024-03-23T21:02:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=65, entry_tid=0' '!=' '' 2024-03-23T21:02:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:02:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=65, entry_tid=0' = '' 2024-03-23T21:02:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=65, entry_tid=0' 2024-03-23T21:02:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=68, entry_tid=0' 2024-03-23T21:02:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:02:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:02:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=68 2024-03-23T21:02:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=65, entry_tid=0' 2024-03-23T21:02:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:02:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:02:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=65 2024-03-23T21:02:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=68, entry_tid=0' 2024-03-23T21:02:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:02:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:02:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-03-23T21:02:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=65, entry_tid=0' 2024-03-23T21:02:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:02:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:02:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-03-23T21:02:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 68 = 65 -a 0 -le 0 2024-03-23T21:02:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:02:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:02:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:02:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:02:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:02:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:02:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:02:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:02:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:02:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:02:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:02:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:02:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=65, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=68, entry_tid=0' 2024-03-23T21:02:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:02:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:02:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:02:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:02:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=65, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=65, entry_tid=0' 2024-03-23T21:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=68, entry_tid=0' -a 'object_number=0, tag_tid=68, entry_tid=0' = 'object_number=0, tag_tid=65, entry_tid=0' 2024-03-23T21:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=65, entry_tid=0' '!=' 'object_number=0, tag_tid=65, entry_tid=0' 2024-03-23T21:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-03-23T21:02:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:02:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:02:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:02:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:02:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:02:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:02:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:02:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:02:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:02:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:02:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:02:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:02:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:02:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:38.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:02:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:02:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:02:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:02:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:02:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=68, entry_tid=0' 2024-03-23T21:02:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:02:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:02:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:02:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:02:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:02:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=68, entry_tid=0' 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=68, entry_tid=0' -a 'object_number=0, tag_tid=68, entry_tid=0' = 'object_number=0, tag_tid=68, entry_tid=0' 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:02:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:37 2024-03-23T21:02:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:02:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2024-03-23T21:02:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:37' 2024-03-23T21:02:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:02:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:02:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:02:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:37' 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:02:37' 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:37' 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:02:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}2024-03-23 21:02:37' 2024-03-23T21:02:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:37' 2024-03-23T21:02:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:02:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:02:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:02:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:02:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:02:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:02:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:02:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2024-03-23T21:02:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:37 2024-03-23T21:02:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:37' 2024-03-23T21:02:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:02:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:02:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2024-03-23T21:02:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:02:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:02:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:37' 2024-03-23T21:02:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:02:37' 2024-03-23T21:02:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:37' 2024-03-23T21:02:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}42894296mirror.0smithi2042024-03-23 21:02:37cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:02:37' 2024-03-23T21:02:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:37' 2024-03-23T21:02:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:02:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:02:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:02:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:02:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:02:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:02:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:02:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:02:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:02:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:02:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:02:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:02:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:02:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:02:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:02:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:02:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:02:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:02:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:02:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:02:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:02:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:44.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:02:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:02:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:02:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:02:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:02:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:02:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:02:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:02:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:02:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:02:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:43 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:42' 2024-03-23T21:02:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:02:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:46 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:47' 2024-03-23T21:02:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:02:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:02:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:46' 2024-03-23T21:02:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:02:46' 2024-03-23T21:02:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:46' 2024-03-23T21:02:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:02:46cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:02:47' 2024-03-23T21:02:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:46' 2024-03-23T21:02:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:02:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:02:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:02:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:47 2024-03-23T21:02:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:02:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:02:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:46' 2024-03-23T21:02:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:02:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:02:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:02:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:47' 2024-03-23T21:02:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:02:47' 2024-03-23T21:02:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:02:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:47' 2024-03-23T21:02:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:02:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:02:46' 2024-03-23T21:02:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:47' 2024-03-23T21:02:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:02:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:02:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:02:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:02:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:02:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:02:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:02:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:02:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:02:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:02:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:02:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:02:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:02:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:02:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:02:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:02:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:02:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:02:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:02:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:02:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:02:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:02:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:02:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:02:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:02:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:02:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:02:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:02:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:02:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:02:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:02:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:02:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:02:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:02:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:02:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:02:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:02:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:02:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:02:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:02:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:02:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:02:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:02:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:02:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:02:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:02:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:02:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:02:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:02:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:02:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:02:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:02:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:02:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:02:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:02:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:02:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:02:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:02:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:02:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:02:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:02:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:02:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:02:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:02:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:02:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:02:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:02:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:02:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:02:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:02:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:02:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:02:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:02:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:02:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:02:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:02:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:02:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:02:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:02:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:02:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:02:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:02:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:02:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:02:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=69, entry_tid=0' 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:02:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:02:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:02:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:02:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:02:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:02:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:02:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:02:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:02:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=69, entry_tid=0' 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=69, entry_tid=0' -a 'object_number=0, tag_tid=69, entry_tid=0' = 'object_number=0, tag_tid=69, entry_tid=0' 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:02:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:02:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:56 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:57' 2024-03-23T21:02:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:02:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:02:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:02:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:02:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:02:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:56' 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:02:56' 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:56' 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:02:56cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}2024-03-23 21:02:57' 2024-03-23T21:02:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:56' 2024-03-23T21:02:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:02:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:02:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:02:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:02:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:02:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:02:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:02:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:02:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:02:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:02:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:57 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:02:56' 2024-03-23T21:02:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:02:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:02:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:02:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2024-03-23T21:02:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:02:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:02:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:02:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:02:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:02:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:02:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:02:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:57' 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:02:57' 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:02:57' 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:02:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:02:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:02:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}44564463mirror.0smithi2042024-03-23 21:02:57cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:02:56' 2024-03-23T21:02:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:02:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:02:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:02:57' 2024-03-23T21:02:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:02:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:02:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:02:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:02:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:02:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:02:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:02:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:02:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:02:59.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:02:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:03:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:03:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:03:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:03:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:03:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:03:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:03:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:03:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:03:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:03:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:01 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:02' 2024-03-23T21:03:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:01' 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:03:01' 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:01' 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:03:01cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:03:02' 2024-03-23T21:03:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:01' 2024-03-23T21:03:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:03:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:02 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:01' 2024-03-23T21:03:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:02' 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:03:02' 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:02' 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:03:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:03:01' 2024-03-23T21:03:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:02' 2024-03-23T21:03:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:03:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:03:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:03:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:03:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:03:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:03:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:03:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:03:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:03:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:03:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:03:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:03:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:03:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:03:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:03:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:03:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:03:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:03:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:03:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:03:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:03:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:03:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:03:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:03:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:03:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:03:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:03:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:03:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:03:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:03:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:03:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:03:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:03:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:03:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:03:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:03:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:03:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:03:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:03:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:03:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:03:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:03:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:03:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:03:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:03:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:03:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:03:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:03:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=73, entry_tid=0' 2024-03-23T21:03:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:03:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:03:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:03:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:03:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:03:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:03:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:03:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:03:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:03:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:03:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:03:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:03:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=73, entry_tid=0' 2024-03-23T21:03:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=73, entry_tid=0' -a 'object_number=0, tag_tid=73, entry_tid=0' = 'object_number=0, tag_tid=73, entry_tid=0' 2024-03-23T21:03:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:03:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:07 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:07' 2024-03-23T21:03:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:03:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:03:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:03:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:03:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:03:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:07' 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:03:07' 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:07' 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:03:07cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}2024-03-23 21:03:07' 2024-03-23T21:03:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:07' 2024-03-23T21:03:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:03:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:03:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:03:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:03:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:03:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:03:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2024-03-23T21:03:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:07 2024-03-23T21:03:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:07' 2024-03-23T21:03:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:03:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:03:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2024-03-23T21:03:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:03:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:03:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:07' 2024-03-23T21:03:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:03:07' 2024-03-23T21:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:07' 2024-03-23T21:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}42894296mirror.0smithi2042024-03-23 21:03:07cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:03:07' 2024-03-23T21:03:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:07' 2024-03-23T21:03:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:03:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:03:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:03:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:03:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:03:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:03:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:03:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:03:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:03:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:03:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:03:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:03:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:03:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:03:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:03:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:03:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:03:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:03:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:03:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:03:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:03:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:03:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:03:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:03:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:03:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:03:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:13 2024-03-23T21:03:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:12' 2024-03-23T21:03:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:03:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:03:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:16 2024-03-23T21:03:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:17' 2024-03-23T21:03:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:16' 2024-03-23T21:03:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:03:16' 2024-03-23T21:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:16' 2024-03-23T21:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:03:16cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:03:17' 2024-03-23T21:03:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:16' 2024-03-23T21:03:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:17 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:16' 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:17' 2024-03-23T21:03:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:03:17' 2024-03-23T21:03:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:17' 2024-03-23T21:03:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:03:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:03:16' 2024-03-23T21:03:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:17' 2024-03-23T21:03:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:03:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:03:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:03:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:03:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:03:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:03:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:03:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:03:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:03:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:03:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:03:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:03:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:03:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:03:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:03:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:03:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:03:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:03:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:03:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:03:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:03:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:03:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:03:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:03:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:03:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:03:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:03:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:03:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:03:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:03:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=74, entry_tid=0' 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:03:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:03:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:03:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:03:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:03:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:03:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:03:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:03:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=74, entry_tid=0' 2024-03-23T21:03:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=74, entry_tid=0' -a 'object_number=0, tag_tid=74, entry_tid=0' = 'object_number=0, tag_tid=74, entry_tid=0' 2024-03-23T21:03:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:03:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:03:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:21 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:22' 2024-03-23T21:03:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:03:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:03:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:03:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:03:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:03:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:21' 2024-03-23T21:03:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:03:21' 2024-03-23T21:03:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:21' 2024-03-23T21:03:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:03:21cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}2024-03-23 21:03:22' 2024-03-23T21:03:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:21' 2024-03-23T21:03:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:03:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:03:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:03:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:03:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:03:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:03:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2024-03-23T21:03:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:22 2024-03-23T21:03:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:21' 2024-03-23T21:03:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:03:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:03:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2024-03-23T21:03:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:03:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:03:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:22' 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:03:22' 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:22' 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}44564463mirror.0smithi2042024-03-23 21:03:22cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:03:21' 2024-03-23T21:03:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:22' 2024-03-23T21:03:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:03:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:03:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:03:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:03:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:03:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:03:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:03:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:03:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:03:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:03:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:26 2024-03-23T21:03:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:27' 2024-03-23T21:03:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:26' 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:03:26' 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:26' 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:28.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:03:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:03:27' 2024-03-23T21:03:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:26' 2024-03-23T21:03:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:03:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:27 2024-03-23T21:03:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:26' 2024-03-23T21:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:27' 2024-03-23T21:03:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:03:27' 2024-03-23T21:03:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:27' 2024-03-23T21:03:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:03:27cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:03:26' 2024-03-23T21:03:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:27' 2024-03-23T21:03:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:03:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:03:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:03:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:03:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:03:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:03:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:03:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:03:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:03:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:03:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:03:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:03:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:03:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:03:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:03:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:03:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:03:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:03:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:03:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:03:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:03:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:03:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:03:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:03:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:03:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:03:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:03:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:03:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:03:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:03:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:03:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:33.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:03:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:03:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:03:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:03:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:03:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:03:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:03:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:03:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:03:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:03:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:03:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:03:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:03:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:03:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:03:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=78, entry_tid=0' 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:03:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:03:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:03:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:03:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:03:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:03:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:03:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:03:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=78, entry_tid=0' 2024-03-23T21:03:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=78, entry_tid=0' -a 'object_number=0, tag_tid=78, entry_tid=0' = 'object_number=0, tag_tid=78, entry_tid=0' 2024-03-23T21:03:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:03:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:32 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:32' 2024-03-23T21:03:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:03:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:03:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:03:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:03:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:03:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:32' 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:03:32' 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:32' 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:03:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}2024-03-23 21:03:32' 2024-03-23T21:03:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:32' 2024-03-23T21:03:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:03:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:03:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:03:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:03:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:03:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:03:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:32 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:32' 2024-03-23T21:03:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:03:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:03:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2024-03-23T21:03:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:03:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:03:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:32' 2024-03-23T21:03:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:03:32' 2024-03-23T21:03:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:32' 2024-03-23T21:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}42894296mirror.0smithi2042024-03-23 21:03:32cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:03:32' 2024-03-23T21:03:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:32' 2024-03-23T21:03:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:03:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:03:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:03:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:03:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:03:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:03:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:03:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:03:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:03:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:03:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:03:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:03:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:03:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:03:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:03:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:03:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:03:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:03:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:03:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:03:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:03:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:03:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:03:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:03:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:03:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:37 2024-03-23T21:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:37' 2024-03-23T21:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:03:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:41 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:37' 2024-03-23T21:03:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:41' 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:03:41' 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:41' 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:03:41cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:03:42' 2024-03-23T21:03:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:41' 2024-03-23T21:03:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:03:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:42 2024-03-23T21:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:41' 2024-03-23T21:03:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:43.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:42' 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:03:42' 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:42' 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:03:42cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:03:41' 2024-03-23T21:03:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:42' 2024-03-23T21:03:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:03:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:03:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:03:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:03:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:46.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:46.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:46.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:03:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:03:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:03:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:03:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:50.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:03:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:03:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:03:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:03:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:03:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:03:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:03:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:03:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:03:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:03:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:03:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:03:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:03:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:03:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:03:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:03:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:03:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:03:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:03:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:03:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:03:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:03:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:03:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:03:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:03:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:03:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:03:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:03:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:03:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:03:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:03:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:03:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=79, entry_tid=0' 2024-03-23T21:03:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:03:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:03:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:03:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:03:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:03:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:03:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:03:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:03:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:03:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:03:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:03:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=79, entry_tid=0' 2024-03-23T21:03:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=79, entry_tid=0' -a 'object_number=0, tag_tid=79, entry_tid=0' = 'object_number=0, tag_tid=79, entry_tid=0' 2024-03-23T21:03:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:52 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:47' 2024-03-23T21:03:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:03:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:03:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:03:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:03:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:03:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:52' 2024-03-23T21:03:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:03:52' 2024-03-23T21:03:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:03:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:52' 2024-03-23T21:03:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:03:52cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}2024-03-23 21:03:47' 2024-03-23T21:03:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:52' 2024-03-23T21:03:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:03:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:03:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:03:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:03:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:03:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:03:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2024-03-23T21:03:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:47 2024-03-23T21:03:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:51' 2024-03-23T21:03:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:03:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:03:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2024-03-23T21:03:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:03:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:03:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:47' 2024-03-23T21:03:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:03:47' 2024-03-23T21:03:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:03:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:47' 2024-03-23T21:03:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}44564463mirror.0smithi2042024-03-23 21:03:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:03:51' 2024-03-23T21:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:47' 2024-03-23T21:03:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:03:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:03:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:03:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:03:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:03:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:03:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:03:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:03:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:03:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:03:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:03:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:03:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:03:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:03:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:03:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:03:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:03:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:03:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:03:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:03:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:03:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:03:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:03:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:03:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:03:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:03:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:03:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:03:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:03:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:03:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:03:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:03:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:03:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:52 2024-03-23T21:03:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:03:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:03:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:55' 2024-03-23T21:03:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:03:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:03:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:03:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:03:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:56 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:57' 2024-03-23T21:03:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:58.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:58.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:58.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:56' 2024-03-23T21:03:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:03:56' 2024-03-23T21:03:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:56' 2024-03-23T21:03:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:03:56cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:03:57' 2024-03-23T21:03:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:56' 2024-03-23T21:03:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:03:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:03:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:03:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:03:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:03:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:03:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:57 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:03:56' 2024-03-23T21:03:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:03:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:03:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:03:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:03:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:03:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:03:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:03:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:03:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:57' 2024-03-23T21:03:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:03:57' 2024-03-23T21:03:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:03:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:03:57' 2024-03-23T21:03:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:03:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:03:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:03:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:03:57cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:03:56' 2024-03-23T21:03:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:03:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:03:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:03:57' 2024-03-23T21:03:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:03:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:03:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:03:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:03:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:03:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:03:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:03:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:03:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:03:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:03:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:03:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:03:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:03:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:04:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:04:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:04:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:04:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:04:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:04:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:04:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:04:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:04:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:04:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:04:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:04:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:04:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:04:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=83, entry_tid=0' 2024-03-23T21:04:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:04:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:04:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:04:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:04:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=83, entry_tid=0' 2024-03-23T21:04:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=83, entry_tid=0' -a 'object_number=0, tag_tid=83, entry_tid=0' = 'object_number=0, tag_tid=83, entry_tid=0' 2024-03-23T21:04:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:04:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:04.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:02 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2024-03-23T21:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:02' 2024-03-23T21:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:04:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:02' 2024-03-23T21:04:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:04:02' 2024-03-23T21:04:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:02' 2024-03-23T21:04:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:04:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}2024-03-23 21:04:02' 2024-03-23T21:04:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:02' 2024-03-23T21:04:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:04:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:04:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:04:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:02 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:02' 2024-03-23T21:04:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:04:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:04:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2024-03-23T21:04:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:04:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:04:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:02' 2024-03-23T21:04:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:04:02' 2024-03-23T21:04:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:02' 2024-03-23T21:04:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}42894296mirror.0smithi2042024-03-23 21:04:02cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:04:02' 2024-03-23T21:04:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:02' 2024-03-23T21:04:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:04:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:04:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:04:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:04:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:04:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:04:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:04:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:04:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:04:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:04:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:04:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:04:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:04:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:04:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:04:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:04:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:04:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:04:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:07 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:07' 2024-03-23T21:04:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:04:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:12 2024-03-23T21:04:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:12' 2024-03-23T21:04:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:04:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:04:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:12' 2024-03-23T21:04:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:04:12' 2024-03-23T21:04:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:12' 2024-03-23T21:04:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:04:12cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:04:12' 2024-03-23T21:04:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:12' 2024-03-23T21:04:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:04:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:04:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:04:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:04:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:12 2024-03-23T21:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:11' 2024-03-23T21:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:04:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:04:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:12' 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:04:12' 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:12' 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:04:12cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:04:11' 2024-03-23T21:04:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:12' 2024-03-23T21:04:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:04:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:04:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:04:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:04:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:04:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:04:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:04:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:04:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:04:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:04:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:04:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:04:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:04:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:04:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:04:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:04:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:04:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:04:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:04:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Starting"' 2024-03-23T21:04:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Starting"' 2024-03-23T21:04:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:04:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:04:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:04:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:04:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:04:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:04:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:04:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:04:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:04:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:04:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:04:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:04:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:04:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:04:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:04:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:04:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:04:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:04:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:04:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:04:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:04:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:04:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:04:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:04:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:04:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:04:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:04:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:04:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:04:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:04:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:04:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:04:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:04:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:04:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:04:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:04:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=84, entry_tid=0' 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:04:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=84, entry_tid=0' 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=84, entry_tid=0' -a 'object_number=0, tag_tid=84, entry_tid=0' = 'object_number=0, tag_tid=84, entry_tid=0' 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:04:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:21 2024-03-23T21:04:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2024-03-23T21:04:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:17' 2024-03-23T21:04:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:04:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:04:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:04:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:21' 2024-03-23T21:04:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:04:21' 2024-03-23T21:04:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:21' 2024-03-23T21:04:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:04:21cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}2024-03-23 21:04:17' 2024-03-23T21:04:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:21' 2024-03-23T21:04:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:04:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:04:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:04:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2024-03-23T21:04:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:17 2024-03-23T21:04:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:21' 2024-03-23T21:04:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:04:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:04:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2024-03-23T21:04:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:04:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:04:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:17' 2024-03-23T21:04:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:04:17' 2024-03-23T21:04:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:17' 2024-03-23T21:04:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}44564463mirror.0smithi2042024-03-23 21:04:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:04:21' 2024-03-23T21:04:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:17' 2024-03-23T21:04:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:04:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:04:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:04:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:04:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:04:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:04:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:04:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:04:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:04:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:04:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:04:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:04:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:04:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:04:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:04:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:04:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:04:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:04:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:04:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:04:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:04:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:27 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:27' 2024-03-23T21:04:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:04:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:04:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:27' 2024-03-23T21:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:04:27' 2024-03-23T21:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:27' 2024-03-23T21:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:04:27cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:04:27' 2024-03-23T21:04:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:27' 2024-03-23T21:04:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:04:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:04:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:04:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:27 2024-03-23T21:04:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:27' 2024-03-23T21:04:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:04:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:04:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:27' 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:04:27' 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:27' 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:04:27cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:04:27' 2024-03-23T21:04:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:27' 2024-03-23T21:04:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:04:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:04:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:04:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:04:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:04:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:04:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:04:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:04:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:04:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:04:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:04:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:04:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:04:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:04:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:04:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:04:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:04:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:04:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:04:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:04:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:04:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:04:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:04:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:04:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:04:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:04:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:04:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:04:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:04:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:04:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:04:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=88, entry_tid=0' 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:04:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:04:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=88, entry_tid=0' 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=88, entry_tid=0' -a 'object_number=0, tag_tid=88, entry_tid=0' = 'object_number=0, tag_tid=88, entry_tid=0' 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:04:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:32 2024-03-23T21:04:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2024-03-23T21:04:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:32' 2024-03-23T21:04:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:04:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:04:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:04:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:32' 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:04:32' 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:32' 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:04:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}2024-03-23 21:04:32' 2024-03-23T21:04:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:32' 2024-03-23T21:04:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:04:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:04:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:04:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:04:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:32 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:32' 2024-03-23T21:04:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:04:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:04:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2024-03-23T21:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:04:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:04:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:32' 2024-03-23T21:04:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:04:32' 2024-03-23T21:04:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:32' 2024-03-23T21:04:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}42894296mirror.0smithi2042024-03-23 21:04:32cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:04:32' 2024-03-23T21:04:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:32' 2024-03-23T21:04:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:04:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:04:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:04:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:04:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:04:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:04:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:04:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:37.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:04:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:04:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:04:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:04:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:04:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:04:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:04:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:04:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:38 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:37' 2024-03-23T21:04:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:04:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:04:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:42 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:42' 2024-03-23T21:04:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:04:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:04:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:42' 2024-03-23T21:04:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:04:42' 2024-03-23T21:04:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:42' 2024-03-23T21:04:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:04:42cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:04:42' 2024-03-23T21:04:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:42' 2024-03-23T21:04:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:04:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:04:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:04:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:43.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:42 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:42' 2024-03-23T21:04:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:42' 2024-03-23T21:04:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:04:42' 2024-03-23T21:04:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:42' 2024-03-23T21:04:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:44.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:04:42cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:04:42' 2024-03-23T21:04:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:42' 2024-03-23T21:04:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:04:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:04:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:04:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:04:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:04:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:04:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:04:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:04:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:04:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:04:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:04:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:04:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:45.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:04:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:04:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:04:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:04:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:04:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:04:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:04:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:04:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:04:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:04:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:04:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:04:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:04:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:04:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:04:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:04:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:04:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:04:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:04:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:04:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:04:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:04:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:04:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:04:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:04:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:04:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:04:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:04:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:04:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:04:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:04:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=86, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=89, entry_tid=0' 2024-03-23T21:04:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:04:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:04:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=86, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=86, entry_tid=0' 2024-03-23T21:04:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=89, entry_tid=0' -a 'object_number=0, tag_tid=89, entry_tid=0' = 'object_number=0, tag_tid=86, entry_tid=0' 2024-03-23T21:04:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=86, entry_tid=0' '!=' '' 2024-03-23T21:04:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:04:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=86, entry_tid=0' = '' 2024-03-23T21:04:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=86, entry_tid=0' 2024-03-23T21:04:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=89, entry_tid=0' 2024-03-23T21:04:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:04:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:04:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=89 2024-03-23T21:04:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=86, entry_tid=0' 2024-03-23T21:04:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:04:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:04:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=86 2024-03-23T21:04:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=89, entry_tid=0' 2024-03-23T21:04:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:04:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:04:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-03-23T21:04:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=86, entry_tid=0' 2024-03-23T21:04:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:04:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:04:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-03-23T21:04:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 89 = 86 -a 0 -le 0 2024-03-23T21:04:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:04:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:04:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:04:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:04:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:04:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:04:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:04:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:04:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:04:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:04:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:04:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:04:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=89, entry_tid=0' 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:04:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:04:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:04:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=89, entry_tid=0' 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=89, entry_tid=0' -a 'object_number=0, tag_tid=89, entry_tid=0' = 'object_number=0, tag_tid=89, entry_tid=0' 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:47 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2024-03-23T21:04:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:47' 2024-03-23T21:04:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:04:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:04:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:04:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:47' 2024-03-23T21:04:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:04:47' 2024-03-23T21:04:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:47' 2024-03-23T21:04:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:04:47cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}2024-03-23 21:04:47' 2024-03-23T21:04:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:47' 2024-03-23T21:04:49.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:04:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:04:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:04:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2024-03-23T21:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:47 2024-03-23T21:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:47' 2024-03-23T21:04:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:04:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:04:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2024-03-23T21:04:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:04:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:47' 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:04:47' 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:47' 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}44564463mirror.0smithi2042024-03-23 21:04:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:04:47' 2024-03-23T21:04:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:47' 2024-03-23T21:04:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:04:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:04:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:04:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:04:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:04:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:04:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:04:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:04:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:04:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:04:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:04:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:04:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:04:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:04:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:04:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:04:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:04:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:04:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:04:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:04:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:04:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:04:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:52 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:52' 2024-03-23T21:04:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:04:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:04:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:52' 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:04:52' 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:52' 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:04:52cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:04:52' 2024-03-23T21:04:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:52' 2024-03-23T21:04:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:04:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:04:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:52 2024-03-23T21:04:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:04:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:04:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:52' 2024-03-23T21:04:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:04:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:04:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:04:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:52' 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:04:52' 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:52' 2024-03-23T21:04:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:04:52cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:04:52' 2024-03-23T21:04:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:52' 2024-03-23T21:04:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:04:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:04:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:04:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:04:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:04:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:04:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:04:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:04:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:04:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:04:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:04:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:04:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:04:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:04:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:04:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:04:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:04:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:04:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:04:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:04:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:04:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:04:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:04:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:04:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:04:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:04:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:04:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:04:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:04:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:04:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:04:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:04:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:04:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:04:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:04:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:04:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:04:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:04:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:04:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:04:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:04:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:04:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:04:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:04:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:04:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:04:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:04:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:04:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:04:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:04:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=93, entry_tid=0' 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:04:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:04:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:04:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:04:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:04:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:04:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:04:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:04:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:04:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=93, entry_tid=0' 2024-03-23T21:04:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=93, entry_tid=0' -a 'object_number=0, tag_tid=93, entry_tid=0' = 'object_number=0, tag_tid=93, entry_tid=0' 2024-03-23T21:04:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:04:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:04:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:04:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:04:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:04:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:04:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:04:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:04:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:57 2024-03-23T21:04:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:04:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:04:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:04:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2024-03-23T21:04:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:57' 2024-03-23T21:04:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:04:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:04:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:04:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:04:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:04:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:04:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:04:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:04:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:04:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:04:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:57' 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:04:57' 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:57' 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:04:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:04:57cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}2024-03-23 21:04:57' 2024-03-23T21:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:04:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:04:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:57' 2024-03-23T21:04:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:04:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:04:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:04:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:04:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:04:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:04:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:05:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:05:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:57 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:04:57' 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:05:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:05:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2024-03-23T21:05:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:05:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:05:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:57' 2024-03-23T21:05:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:04:57' 2024-03-23T21:05:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:04:57' 2024-03-23T21:05:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}42894296mirror.0smithi2042024-03-23 21:04:57cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:04:57' 2024-03-23T21:05:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:04:57' 2024-03-23T21:05:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:05:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:05:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:05:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:05:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:05:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:05:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:05:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:05:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:05:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:05:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:05:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:05:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:05:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:05:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:05:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:05:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:05:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:05:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:05:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:05:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:05:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:05:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:05:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:05:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:05:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:05:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:05:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:05:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:05:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:05:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:05:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:02 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:02' 2024-03-23T21:05:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:05:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:05:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:07 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:07' 2024-03-23T21:05:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:05:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:05:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:07' 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:05:07' 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:07' 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:05:07cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:05:07' 2024-03-23T21:05:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:07' 2024-03-23T21:05:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:05:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:05:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:05:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:05:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:07 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:07' 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:05:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:05:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:08.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:07' 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:05:07' 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:07' 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:05:07cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:05:07' 2024-03-23T21:05:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:07' 2024-03-23T21:05:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:05:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:05:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:05:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:05:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:05:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:05:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:05:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:05:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:05:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:05:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:05:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:05:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:05:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:05:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:05:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:16.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:05:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:05:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:05:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:05:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:05:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:05:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:05:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:05:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:05:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:05:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:05:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:05:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:05:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:05:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:05:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:05:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:05:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:05:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:05:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:05:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:05:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=94, entry_tid=0' 2024-03-23T21:05:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:05:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:05:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:05:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:05:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:05:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:05:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:05:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:05:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:05:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:05:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:05:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:05:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=94, entry_tid=0' 2024-03-23T21:05:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=94, entry_tid=0' -a 'object_number=0, tag_tid=94, entry_tid=0' = 'object_number=0, tag_tid=94, entry_tid=0' 2024-03-23T21:05:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:05:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:05:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:17 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:05:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2024-03-23T21:05:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:12' 2024-03-23T21:05:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:05:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:05:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:05:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:05:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:05:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:17' 2024-03-23T21:05:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:05:17' 2024-03-23T21:05:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:17' 2024-03-23T21:05:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:05:17cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}2024-03-23 21:05:12' 2024-03-23T21:05:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:17' 2024-03-23T21:05:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:05:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:05:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:05:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:12 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:05:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:17' 2024-03-23T21:05:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:05:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:05:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2024-03-23T21:05:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:05:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:05:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:12' 2024-03-23T21:05:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:05:12' 2024-03-23T21:05:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:12' 2024-03-23T21:05:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}44564463mirror.0smithi2042024-03-23 21:05:12cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:05:17' 2024-03-23T21:05:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:12' 2024-03-23T21:05:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:05:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:05:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:05:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:05:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:05:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:05:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:05:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:05:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:05:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:05:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:05:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:05:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:05:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:20.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:05:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:05:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:05:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:05:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:17 2024-03-23T21:05:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:05:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:20' 2024-03-23T21:05:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:05:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:22 2024-03-23T21:05:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:22' 2024-03-23T21:05:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:05:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:05:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:22' 2024-03-23T21:05:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:05:22' 2024-03-23T21:05:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:22' 2024-03-23T21:05:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:05:22cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:05:22' 2024-03-23T21:05:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:22' 2024-03-23T21:05:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:05:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:05:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:24.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:05:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:05:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:22 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:22' 2024-03-23T21:05:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:05:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:05:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:22' 2024-03-23T21:05:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:05:22' 2024-03-23T21:05:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:22' 2024-03-23T21:05:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:05:22cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:05:22' 2024-03-23T21:05:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:22' 2024-03-23T21:05:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:05:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:05:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:05:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:05:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:05:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:05:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:05:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:05:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:05:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:05:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:05:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:05:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:05:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:05:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:05:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:05:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:05:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:05:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:05:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:05:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:05:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:05:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:05:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:05:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:05:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:05:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:05:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:05:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:05:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:05:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:05:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:05:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:05:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:05:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:05:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:05:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:05:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:05:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:05:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:05:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:05:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:05:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:05:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:05:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:05:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:05:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:05:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=98, entry_tid=0' 2024-03-23T21:05:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:05:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:05:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:05:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:05:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:05:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:05:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:05:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:05:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:05:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:05:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:05:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=98, entry_tid=0' 2024-03-23T21:05:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=98, entry_tid=0' -a 'object_number=0, tag_tid=98, entry_tid=0' = 'object_number=0, tag_tid=98, entry_tid=0' 2024-03-23T21:05:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:05:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:05:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:05:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:05:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:27 2024-03-23T21:05:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:05:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:05:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2024-03-23T21:05:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:27' 2024-03-23T21:05:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:05:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:05:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:05:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:05:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:27' 2024-03-23T21:05:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:05:27' 2024-03-23T21:05:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:27' 2024-03-23T21:05:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:05:27cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}2024-03-23 21:05:27' 2024-03-23T21:05:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:27' 2024-03-23T21:05:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:05:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:05:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:05:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:05:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:05:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:05:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:27 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:27' 2024-03-23T21:05:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:05:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:05:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2024-03-23T21:05:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:05:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:05:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:27' 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:05:27' 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:27' 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}42894296mirror.0smithi2042024-03-23 21:05:27cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:05:27' 2024-03-23T21:05:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:27' 2024-03-23T21:05:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:05:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:05:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:05:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:05:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:05:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:05:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:05:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:05:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:05:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:05:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:05:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:05:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:32 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:32' 2024-03-23T21:05:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:05:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:05:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:37 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:37' 2024-03-23T21:05:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:05:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:05:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:37' 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:05:37' 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:37' 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:05:37cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:05:37' 2024-03-23T21:05:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:37' 2024-03-23T21:05:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:05:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:05:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:05:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:05:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:37 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:37' 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:05:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:05:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:37' 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:05:37' 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:37' 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:05:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:05:37' 2024-03-23T21:05:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:37' 2024-03-23T21:05:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:05:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:05:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:05:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:05:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:05:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:05:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:05:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:05:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:05:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:05:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:05:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:05:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:05:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:05:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:05:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:05:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:05:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:05:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:05:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:05:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:05:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:05:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:05:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:05:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:05:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:05:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:05:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:05:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:05:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:05:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:05:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:05:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:05:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:05:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:05:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:05:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:05:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:05:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:05:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:05:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:05:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:05:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:05:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:05:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:05:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:05:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:05:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:05:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:05:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:05:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:05:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=99, entry_tid=0' 2024-03-23T21:05:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:05:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:05:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:05:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:05:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:05:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=99, entry_tid=0' 2024-03-23T21:05:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=99, entry_tid=0' -a 'object_number=0, tag_tid=99, entry_tid=0' = 'object_number=0, tag_tid=99, entry_tid=0' 2024-03-23T21:05:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:05:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:47 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:05:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2024-03-23T21:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:42' 2024-03-23T21:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:05:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:05:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:05:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:05:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:47' 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:05:47' 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:47' 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:05:47cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}2024-03-23 21:05:42' 2024-03-23T21:05:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:47' 2024-03-23T21:05:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:05:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:05:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:05:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:05:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:05:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:05:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:05:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:42 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:46' 2024-03-23T21:05:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:05:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:05:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2024-03-23T21:05:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:05:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:05:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:42' 2024-03-23T21:05:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:05:42' 2024-03-23T21:05:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:05:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:42' 2024-03-23T21:05:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}44564463mirror.0smithi2042024-03-23 21:05:42cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:05:46' 2024-03-23T21:05:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:42' 2024-03-23T21:05:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:05:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:05:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:05:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:05:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:05:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:05:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:05:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:05:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:05:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:05:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:05:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:05:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:05:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:05:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:05:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:05:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:05:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:05:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:05:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:05:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:05:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:05:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:05:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:05:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:05:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:05:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:05:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:05:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:05:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:05:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:05:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:05:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:05:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:05:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:52 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:52' 2024-03-23T21:05:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:05:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:05:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:52' 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:05:52' 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:52' 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:05:52cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:05:52' 2024-03-23T21:05:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:52' 2024-03-23T21:05:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:05:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:05:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:05:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:05:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:05:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:05:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:05:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:52 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:52' 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:05:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:05:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:05:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:05:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:05:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:05:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:05:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:05:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:05:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:05:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:52' 2024-03-23T21:05:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:05:52' 2024-03-23T21:05:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:05:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:52' 2024-03-23T21:05:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:05:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:05:52cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:05:52' 2024-03-23T21:05:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:05:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:05:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:52' 2024-03-23T21:05:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:05:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:05:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:05:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:05:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:05:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:05:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:05:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:05:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:05:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:05:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:05:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:05:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:05:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:05:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:05:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:05:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:05:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:05:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:05:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:05:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:05:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:05:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:05:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:05:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:05:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:05:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:05:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:05:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:05:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:05:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:05:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:05:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:05:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:05:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:05:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:05:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:05:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:05:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:05:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:05:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:05:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:05:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:05:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:05:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:05:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:05:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:05:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:05:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:05:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:05:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:05:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:05:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:05:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:05:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:05:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:05:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:05:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:05:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:05:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:05:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:05:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:05:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:05:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:05:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:05:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:05:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:05:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:05:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:05:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:05:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=103, entry_tid=0' 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:05:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:05:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:05:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:05:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:05:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:05:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:05:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:05:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=103, entry_tid=0' 2024-03-23T21:05:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=103, entry_tid=0' -a 'object_number=0, tag_tid=103, entry_tid=0' = 'object_number=0, tag_tid=103, entry_tid=0' 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:05:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:05:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:05:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:05:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:05:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:05:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:05:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:05:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:05:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:05:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:05:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:57 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:57' 2024-03-23T21:06:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:06:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:06:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:06:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:57' 2024-03-23T21:06:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:05:57' 2024-03-23T21:06:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:57' 2024-03-23T21:06:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:05:57cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}2024-03-23 21:05:57' 2024-03-23T21:06:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:57' 2024-03-23T21:06:00.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:06:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:06:00.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:06:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2024-03-23T21:06:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:57 2024-03-23T21:06:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:05:57' 2024-03-23T21:06:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:06:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:06:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2024-03-23T21:06:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:57' 2024-03-23T21:06:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:05:57' 2024-03-23T21:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:05:57' 2024-03-23T21:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}42894296mirror.0smithi2042024-03-23 21:05:57cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:05:57' 2024-03-23T21:06:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:05:57' 2024-03-23T21:06:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:06:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:06:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:06:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:06:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:06:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:06:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:06:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:06:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:04.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:06:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:03 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:02' 2024-03-23T21:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:06:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:06:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:07 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:07' 2024-03-23T21:06:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:06:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:06:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:07' 2024-03-23T21:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:06:07' 2024-03-23T21:06:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:07' 2024-03-23T21:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:06:07cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:06:07' 2024-03-23T21:06:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:07' 2024-03-23T21:06:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:06:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:06:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:07 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:07' 2024-03-23T21:06:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:06:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:06:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:07' 2024-03-23T21:06:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:06:07' 2024-03-23T21:06:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:07' 2024-03-23T21:06:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:06:07cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:06:07' 2024-03-23T21:06:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:09.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:07' 2024-03-23T21:06:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:06:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:06:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:06:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:06:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:06:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:06:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:06:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:06:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:06:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:06:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:06:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:06:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:06:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:06:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:06:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:06:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:06:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:06:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:06:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:06:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:06:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:06:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:06:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:06:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:06:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:06:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:06:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:06:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=101, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=104, entry_tid=0' 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:06:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=101, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=101, entry_tid=0' 2024-03-23T21:06:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=104, entry_tid=0' -a 'object_number=0, tag_tid=104, entry_tid=0' = 'object_number=0, tag_tid=101, entry_tid=0' 2024-03-23T21:06:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=101, entry_tid=0' '!=' '' 2024-03-23T21:06:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:06:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=101, entry_tid=0' = '' 2024-03-23T21:06:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=101, entry_tid=0' 2024-03-23T21:06:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=104, entry_tid=0' 2024-03-23T21:06:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:06:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:06:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=104 2024-03-23T21:06:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=101, entry_tid=0' 2024-03-23T21:06:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:06:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:06:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=101 2024-03-23T21:06:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=104, entry_tid=0' 2024-03-23T21:06:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:06:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:06:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-03-23T21:06:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=101, entry_tid=0' 2024-03-23T21:06:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:06:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:06:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-03-23T21:06:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 104 = 101 -a 0 -le 0 2024-03-23T21:06:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:06:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:06:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:06:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:06:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:06:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:06:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:06:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:06:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:06:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:06:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:06:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:06:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:06:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:06:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:06:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:06:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:06:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:06:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:06:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=104, entry_tid=0' 2024-03-23T21:06:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:06:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:06:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:06:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:06:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=104, entry_tid=0' 2024-03-23T21:06:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=104, entry_tid=0' -a 'object_number=0, tag_tid=104, entry_tid=0' = 'object_number=0, tag_tid=104, entry_tid=0' 2024-03-23T21:06:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:06:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:06:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:12 2024-03-23T21:06:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2024-03-23T21:06:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:12' 2024-03-23T21:06:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:06:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:06:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:06:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:12' 2024-03-23T21:06:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:06:12' 2024-03-23T21:06:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:12' 2024-03-23T21:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:06:12cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}2024-03-23 21:06:12' 2024-03-23T21:06:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:12' 2024-03-23T21:06:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:06:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:06:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2024-03-23T21:06:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:12 2024-03-23T21:06:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:12' 2024-03-23T21:06:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:06:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:06:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2024-03-23T21:06:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:12' 2024-03-23T21:06:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:06:12' 2024-03-23T21:06:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:12' 2024-03-23T21:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}44564463mirror.0smithi2042024-03-23 21:06:12cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:06:12' 2024-03-23T21:06:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:12' 2024-03-23T21:06:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:06:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:06:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:06:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:06:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:06:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:06:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:06:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:06:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:06:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:06:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:06:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:06:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:06:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:06:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:06:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:06:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:06:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:06:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:06:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:06:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:06:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:06:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:06:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:17 2024-03-23T21:06:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:06:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:17' 2024-03-23T21:06:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:06:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:06:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:17' 2024-03-23T21:06:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:06:17' 2024-03-23T21:06:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:17' 2024-03-23T21:06:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:06:17cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedstopped2024-03-23 21:06:17' 2024-03-23T21:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:17' 2024-03-23T21:06:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:06:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:06:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:06:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:06:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:17 2024-03-23T21:06:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:17' 2024-03-23T21:06:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:06:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:06:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:22 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:22' 2024-03-23T21:06:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:06:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:06:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:22' 2024-03-23T21:06:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:06:22' 2024-03-23T21:06:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:22' 2024-03-23T21:06:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:06:22cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:06:22' 2024-03-23T21:06:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:22' 2024-03-23T21:06:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:06:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:06:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:06:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:06:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:06:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:06:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:06:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:06:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:06:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:06:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:06:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:06:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:06:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:06:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:06:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:06:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:06:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:06:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:06:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:06:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:06:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:06:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:06:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:06:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:06:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:06:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=108, entry_tid=0' 2024-03-23T21:06:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:06:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:06:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:06:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:06:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:06:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:06:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=108, entry_tid=0' 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=108, entry_tid=0' -a 'object_number=0, tag_tid=108, entry_tid=0' = 'object_number=0, tag_tid=108, entry_tid=0' 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:06:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:27 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2024-03-23T21:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:27' 2024-03-23T21:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:27' 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:06:27' 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:27' 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:06:27cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}2024-03-23 21:06:27' 2024-03-23T21:06:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:27' 2024-03-23T21:06:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:06:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:06:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:06:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2024-03-23T21:06:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:27 2024-03-23T21:06:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:27' 2024-03-23T21:06:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:06:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:06:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2024-03-23T21:06:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:27' 2024-03-23T21:06:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:06:27' 2024-03-23T21:06:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:27' 2024-03-23T21:06:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}42894296mirror.0smithi2042024-03-23 21:06:27cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:06:27' 2024-03-23T21:06:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:27' 2024-03-23T21:06:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:06:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:06:30.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:06:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:06:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:30.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:06:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:31.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:06:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:06:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:06:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:06:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:32 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:32' 2024-03-23T21:06:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:06:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:06:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:32' 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:06:32' 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:32' 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:06:32cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:06:32' 2024-03-23T21:06:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:32' 2024-03-23T21:06:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:06:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:06:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:06:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:32 2024-03-23T21:06:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:32' 2024-03-23T21:06:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:06:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:06:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:32' 2024-03-23T21:06:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:06:32' 2024-03-23T21:06:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:32' 2024-03-23T21:06:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:06:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:06:32' 2024-03-23T21:06:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:32' 2024-03-23T21:06:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:06:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:06:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:06:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:06:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:06:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:06:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:06:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:06:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:06:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:06:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:06:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:06:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:06:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:06:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:06:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:06:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:06:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:06:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:06:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:06:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:06:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:06:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:06:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:06:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:06:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:06:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:06:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:06:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:06:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:06:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:06:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:06:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:06:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=109, entry_tid=0' 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:06:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:06:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:06:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=109, entry_tid=0' 2024-03-23T21:06:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=109, entry_tid=0' -a 'object_number=0, tag_tid=109, entry_tid=0' = 'object_number=0, tag_tid=109, entry_tid=0' 2024-03-23T21:06:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:06:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:06:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:37 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:37' 2024-03-23T21:06:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:06:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:06:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:06:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:37' 2024-03-23T21:06:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:06:37' 2024-03-23T21:06:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:37' 2024-03-23T21:06:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:06:37cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}2024-03-23 21:06:37' 2024-03-23T21:06:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:37' 2024-03-23T21:06:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:06:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:06:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:06:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:06:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:37 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:42' 2024-03-23T21:06:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:06:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:06:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2024-03-23T21:06:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:37' 2024-03-23T21:06:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:06:37' 2024-03-23T21:06:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:37' 2024-03-23T21:06:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}44564463mirror.0smithi2042024-03-23 21:06:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:06:42' 2024-03-23T21:06:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:37' 2024-03-23T21:06:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:06:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:06:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:06:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:06:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:06:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:06:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:06:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:06:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:06:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:06:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:06:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:06:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:06:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:06:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:06:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:06:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:06:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:06:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:06:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:06:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:06:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:06:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:06:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:06:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:06:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:42 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:44' 2024-03-23T21:06:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:06:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:06:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:47 2024-03-23T21:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:47' 2024-03-23T21:06:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:06:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:06:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:47' 2024-03-23T21:06:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:06:47' 2024-03-23T21:06:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:47' 2024-03-23T21:06:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:06:47cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:06:47' 2024-03-23T21:06:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:47' 2024-03-23T21:06:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:06:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:06:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:06:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:06:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:47 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:06:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:06:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:47' 2024-03-23T21:06:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:06:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:06:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:06:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:47' 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:06:47' 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:47' 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:06:47cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:06:47' 2024-03-23T21:06:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:47' 2024-03-23T21:06:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:06:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:06:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:06:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:06:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:06:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:06:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:06:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:06:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:06:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:06:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:06:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:06:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:06:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:06:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:06:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:06:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:06:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:06:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:06:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:06:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:06:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:06:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:06:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:06:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:06:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:06:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:06:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:06:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:06:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:06:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:06:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=113, entry_tid=0' 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:06:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:06:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:06:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:06:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:06:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:06:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:06:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=113, entry_tid=0' 2024-03-23T21:06:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=113, entry_tid=0' -a 'object_number=0, tag_tid=113, entry_tid=0' = 'object_number=0, tag_tid=113, entry_tid=0' 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:06:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:52 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:52' 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:06:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:06:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:06:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:52' 2024-03-23T21:06:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:06:52' 2024-03-23T21:06:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:06:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:52' 2024-03-23T21:06:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:06:52cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}2024-03-23 21:06:52' 2024-03-23T21:06:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:52' 2024-03-23T21:06:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:06:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:06:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:06:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:06:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:06:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:06:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:06:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:52 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:52' 2024-03-23T21:06:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:06:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2024-03-23T21:06:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:06:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:06:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:06:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:06:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:52' 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:06:52' 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:06:52' 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:06:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:06:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}42894296mirror.0smithi2042024-03-23 21:06:52cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:06:52' 2024-03-23T21:06:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:06:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:06:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:06:52' 2024-03-23T21:06:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:06:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:06:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:06:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:06:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:06:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:06:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-03-23T21:06:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:06:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-03-23T21:06:56.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:06:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:06:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:06:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:06:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:06:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:06:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:06:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:06:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:06:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:06:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:06:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:06:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:06:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:06:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:06:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:06:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:06:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:06:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:06:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:06:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:06:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:06:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:06:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:06:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:06:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:06:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:06:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:06:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:06:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:06:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:06:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:06:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:06:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:06:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:06:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:06:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:06:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:06:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:06:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:06:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:06:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:06:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:06:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:06:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:06:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:06:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:06:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:06:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:06:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:06:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:06:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:06:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:07:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:00.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:07:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:07:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:58 2024-03-23T21:07:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:06:57' 2024-03-23T21:07:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:07:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:07:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:07:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:07:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:07:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:02 2024-03-23T21:07:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:02' 2024-03-23T21:07:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:07:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:07:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:07:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:07:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:07:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:02' 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:07:02' 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:02' 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:07:02cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+unknownremote image is not primary2024-03-23 21:07:02' 2024-03-23T21:07:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:03.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:07:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:02' 2024-03-23T21:07:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:07:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:07:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:07:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:07:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:07:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:02 2024-03-23T21:07:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:07:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:02' 2024-03-23T21:07:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:07:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:07:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:07:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:07:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:07:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:02' 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:07:02' 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:02' 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:07:02cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:07:02' 2024-03-23T21:07:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:07:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:02' 2024-03-23T21:07:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:07:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:07:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:07:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:07:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:07:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-03-23T21:07:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:07:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-03-23T21:07:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:07:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-03-23T21:07:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:07:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:07:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:07:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:07:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:07:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:07:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:07:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:07:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:07:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:07:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-03-23T21:07:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:07:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:07:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:07:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:07:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:07:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:07:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:07:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:07:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:07:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:07:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:07:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:07:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:07:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:07:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:07:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:07:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:07:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:07:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:07:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:07:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:07:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:07:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:07:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:07:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:07:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:07:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:07:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:07:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:07:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:07:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:07:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:07:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:07:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:07:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:07:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:07:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=111, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:07:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=114, entry_tid=0' 2024-03-23T21:07:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:07:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:07:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:07:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:07:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=111, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:07:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=111, entry_tid=0' 2024-03-23T21:07:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=114, entry_tid=0' -a 'object_number=0, tag_tid=114, entry_tid=0' = 'object_number=0, tag_tid=111, entry_tid=0' 2024-03-23T21:07:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=111, entry_tid=0' '!=' '' 2024-03-23T21:07:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:07:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=111, entry_tid=0' = '' 2024-03-23T21:07:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=111, entry_tid=0' 2024-03-23T21:07:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=114, entry_tid=0' 2024-03-23T21:07:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:07:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:07:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=114 2024-03-23T21:07:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=111, entry_tid=0' 2024-03-23T21:07:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:07:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:07:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=111 2024-03-23T21:07:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=114, entry_tid=0' 2024-03-23T21:07:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:07:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:07:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-03-23T21:07:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=111, entry_tid=0' 2024-03-23T21:07:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:07:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:07:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-03-23T21:07:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 114 = 111 -a 0 -le 0 2024-03-23T21:07:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:07:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:07:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:07:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-03-23T21:07:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:07:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:07:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:07:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-03-23T21:07:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:07:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:07:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:07:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:07:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:07:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:07:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:07:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:07:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:07:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:07:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:07:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:07:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:07:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:07:08.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=114, entry_tid=0' 2024-03-23T21:07:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:07:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-03-23T21:07:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:07:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=e9ef87d4-e149-495b-ba3d-134c4a685598, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-03-23T21:07:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=114, entry_tid=0' 2024-03-23T21:07:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=114, entry_tid=0' -a 'object_number=0, tag_tid=114, entry_tid=0' = 'object_number=0, tag_tid=114, entry_tid=0' 2024-03-23T21:07:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-03-23T21:07:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-03-23T21:07:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:07:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:07 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:07' 2024-03-23T21:07:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:07:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:07:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:07:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:07' 2024-03-23T21:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-03-23 21:07:07' 2024-03-23T21:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:07' 2024-03-23T21:07:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary42894296mirror.0smithi2042024-03-23 21:07:07cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}2024-03-23 21:07:07' 2024-03-23T21:07:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:07:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:07' 2024-03-23T21:07:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:07:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:07:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:07:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-03-23T21:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:07:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:07 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:07' 2024-03-23T21:07:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:07:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:07:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2024-03-23T21:07:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:07' 2024-03-23T21:07:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-03-23 21:07:07' 2024-03-23T21:07:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:07' 2024-03-23T21:07:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}44564463mirror.0smithi2042024-03-23 21:07:07cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:07:07' 2024-03-23T21:07:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:07:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:07' 2024-03-23T21:07:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:07:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:07:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:07:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-03-23T21:07:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-03-23T21:07:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:07:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:07:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:07:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:07:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:07:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:07:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:07:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:07:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:07:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:07:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-03-23T21:07:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:07:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-03-23T21:07:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:07:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-03-23T21:07:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:07:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:07:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:07:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:07:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:07:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:07:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-03-23T21:07:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-03-23T21:07:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:12 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:07:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:12' 2024-03-23T21:07:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:07:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:07:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:07:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:07:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:07:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:12' 2024-03-23T21:07:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-03-23 21:07:12' 2024-03-23T21:07:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:12' 2024-03-23T21:07:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary42894296mirror.0smithi2042024-03-23 21:07:12cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedstopped2024-03-23 21:07:12' 2024-03-23T21:07:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:07:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:12' 2024-03-23T21:07:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:07:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:07:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:07:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:07:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:07:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-03-23T21:07:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:07:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:12 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:12' 2024-03-23T21:07:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:07:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:07:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:07:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-03-23T21:07:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:07:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:17 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-03-23T21:07:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:17' 2024-03-23T21:07:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-03-23T21:07:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-03-23T21:07:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-03-23T21:07:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:07:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:07:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:17' 2024-03-23T21:07:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-03-23 21:07:17' 2024-03-23T21:07:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-03-23T21:07:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:17' 2024-03-23T21:07:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+unknownremote image is not primary44564463mirror.0smithi2042024-03-23 21:07:17cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+unknownremote image is not primary2024-03-23 21:07:17' 2024-03-23T21:07:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:17' 2024-03-23T21:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:07:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-03-23T21:07:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-03-23T21:07:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-03-23T21:07:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-03-23T21:07:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-03-23T21:07:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-03-23T21:07:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-03-23T21:07:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:07:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:07:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:07:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:07:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:07:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:07:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:07:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:07:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-03-23T21:07:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:07:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:07:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:07:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:07:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:07:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:07:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:07:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:07:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:07:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:07:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:07:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:07:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:07:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-03-23T21:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-03-23T21:07:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-03-23T21:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:07:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:07:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:07:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:07:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:07:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:07:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:07:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-03-23T21:07:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-03-23T21:07:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-03-23T21:07:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:07:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:07:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:07:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:07:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:07:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:07:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=118, entry_tid=0' 2024-03-23T21:07:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-03-23T21:07:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-03-23T21:07:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-03-23T21:07:22.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:07:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-03-23T21:07:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.status 2024-03-23T21:07:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=118, entry_tid=0' 2024-03-23T21:07:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=118, entry_tid=0' -a 'object_number=0, tag_tid=118, entry_tid=0' = 'object_number=0, tag_tid=118, entry_tid=0' 2024-03-23T21:07:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-03-23T21:07:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-03-23T21:07:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-03-23T21:07:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:22 2024-03-23T21:07:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:07:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:07:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2024-03-23T21:07:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:22' 2024-03-23T21:07:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:07:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:07:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:07:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:22' 2024-03-23T21:07:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-03-23 21:07:22' 2024-03-23T21:07:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:22' 2024-03-23T21:07:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:07:22cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2024-03-23 21:07:22' 2024-03-23T21:07:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:07:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:22' 2024-03-23T21:07:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:07:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:07:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:07:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-03-23T21:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-03-23T21:07:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:07:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:07:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a690f084-85c3-4b19-8ea7-4e4ff8d04c66 2024-03-23T21:07:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:07:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2024-03-23T21:07:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:22 2024-03-23T21:07:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:22' 2024-03-23T21:07:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:07:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:07:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2024-03-23T21:07:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:22' 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:07:22' 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:22' 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42894296mirror.0smithi2042024-03-23 21:07:22cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:07:22' 2024-03-23T21:07:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:07:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:22' 2024-03-23T21:07:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:07:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:07:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: force promote' 2024-03-23T21:07:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:07:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:07:25 TEST: force promote 2024-03-23T21:07:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:07:25 TEST: force promote 2024-03-23T21:07:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ force_promote_image=test_force_promote 2024-03-23T21:07:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test_force_promote 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_force_promote 2024-03-23T21:07:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_force_promote 100 2024-03-23T21:07:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:07:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:07:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:07:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:07:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:07:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:07:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:07:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.36 bytes/sec: 8.1 MiB/s 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:07:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2024-03-23T21:07:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:07:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:07:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:07:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:07:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:1 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-03-23T21:07:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:07:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:07:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:1 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-03-23T21:07:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:07:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:07:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:07:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:07:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_force_promote 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote started 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_force_promote started 2024-03-23T21:07:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:07:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:07:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:07:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:07:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_force_promote 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:07:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_force_promote 2024-03-23T21:07:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:07:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:07:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:07:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:07:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:07:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:07:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_force_promote 2024-03-23T21:07:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_force_promote ']' 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_force_promote' 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:07:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:07:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:07:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:07:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:07:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:07:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote 2024-03-23T21:07:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:07:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_force_promote 2024-03-23T21:07:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:07:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:07:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:07:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:07:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:07:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:07:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:07:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:07:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:07:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:07:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:30.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_force_promote 2024-03-23T21:07:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_force_promote 2024-03-23T21:07:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-03-23T21:07:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '' 2024-03-23T21:07:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-03-23T21:07:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:07:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-03-23T21:07:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:07:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:07:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '..*' 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-03-23T21:07:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:07:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-03-23T21:07:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:07:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:07:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_force_promote.status 2024-03-23T21:07:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:07:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:07:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position 2024-03-23T21:07:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:07:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position '' 2024-03-23T21:07:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:07:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote 2024-03-23T21:07:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_force_promote: 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 52ae9a17-e4d3-42b0-8bc8-9bd7d1277964 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:29 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:27' 2024-03-23T21:07:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:07:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:07:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:07:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:07:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:07:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:29' 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+replaying '2024-03-23 21:07:29' 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:29' 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42894296mirror.0smithi2042024-03-23 21:07:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:07:27test_force_promote52ae9a17-e4d3-42b0-8bc8-9bd7d1277964up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42934299mirror.1smithi2042024-03-23 21:07:29cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:07:27' 2024-03-23T21:07:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2024-03-23T21:07:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:29' 2024-03-23T21:07:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2024-03-23T21:07:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:07:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:07:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' '' 2024-03-23T21:07:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_force_promote: 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 52ae9a17-e4d3-42b0-8bc8-9bd7d1277964 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:32 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:07:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:07:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:29' 2024-03-23T21:07:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:07:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:07:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:07:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:07:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:32' 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2024-03-23 21:07:32' 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:32' 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK244564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:07:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2024-03-23 21:07:26test_force_promote52ae9a17-e4d3-42b0-8bc8-9bd7d1277964up+stoppedlocal image is primary44604469mirror.1smithi2042024-03-23 21:07:32cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}2024-03-23 21:07:29' 2024-03-23T21:07:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2024-03-23T21:07:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:32' 2024-03-23T21:07:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2024-03-23T21:07:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:07:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:07:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test_force_promote --force 2024-03-23T21:07:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force=--force 2024-03-23T21:07:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_force_promote --force 2024-03-23T21:07:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:07:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test_force_promote 2024-03-23T21:07:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote stopped 2024-03-23T21:07:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:07:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_force_promote stopped 2024-03-23T21:07:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:07:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:07:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:07:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:07:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:07:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2024-03-23T21:07:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:07:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-03-23T21:07:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-03-23T21:07:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:07:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:35.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:07:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:07:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:07:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-03-23T21:07:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:07:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:07:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:07:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:07:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:1 2024-03-23T21:07:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:07:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:07:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-03-23T21:07:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:07:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:07:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:07:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+stopped 2024-03-23T21:07:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+stopped '' '' 2024-03-23T21:07:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote 2024-03-23T21:07:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_force_promote: 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 52ae9a17-e4d3-42b0-8bc8-9bd7d1277964 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:37 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:32' 2024-03-23T21:07:37.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:07:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:07:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:07:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:37.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:07:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:37' 2024-03-23T21:07:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+stopped '2024-03-23 21:07:37' 2024-03-23T21:07:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:37' 2024-03-23T21:07:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK1142894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42894296mirror.0smithi2042024-03-23 21:07:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:07:32test_force_promote52ae9a17-e4d3-42b0-8bc8-9bd7d1277964up+stoppedlocal image is primary42934299mirror.1smithi2042024-03-23 21:07:37cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:07:32' 2024-03-23T21:07:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2024-03-23T21:07:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:37' 2024-03-23T21:07:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2024-03-23T21:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:07:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:07:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' '' 2024-03-23T21:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote 2024-03-23T21:07:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_force_promote: 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 52ae9a17-e4d3-42b0-8bc8-9bd7d1277964 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:37 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:37' 2024-03-23T21:07:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:07:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:07:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:07:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-03-23T21:07:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:07:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:37' 2024-03-23T21:07:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2024-03-23 21:07:37' 2024-03-23T21:07:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:07:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:37' 2024-03-23T21:07:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK1144564463mirror.0smithi20418.2.2-954-g82316b04trueOK44604469mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+stoppedlocal image is primary44564463mirror.0smithi2042024-03-23 21:07:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2024-03-23 21:07:26test_force_promote52ae9a17-e4d3-42b0-8bc8-9bd7d1277964up+stoppedlocal image is primary44604469mirror.1smithi2042024-03-23 21:07:37cluster1b3e423e8-28b3-4966-87da-be3892bfa4caup+stoppedlocal image is primary2024-03-23 21:07:37' 2024-03-23T21:07:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2024-03-23T21:07:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:37' 2024-03-23T21:07:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2024-03-23T21:07:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:07:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:07:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:07:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster1 mirror test_force_promote 100 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:07:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:07:38.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:07:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:07:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:07:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1785.74 bytes/sec: 7.0 MiB/s 2024-03-23T21:07:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_force_promote 100 2024-03-23T21:07:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:07:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:07:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:07:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:07:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:07:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:07:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:07:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.36 bytes/sec: 8.1 MiB/s 2024-03-23T21:07:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster1 mirror test_force_promote 2024-03-23T21:07:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:07:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster1 mirror test_force_promote 2024-03-23T21:07:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_force_promote 2024-03-23T21:07:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 rm mirror/test_force_promote 2024-03-23T21:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:07:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_force_promote 2024-03-23T21:07:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:07:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_force_promote 2024-03-23T21:07:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-03-23T21:07:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_force_promote 2024-03-23T21:07:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_force_promote 2024-03-23T21:07:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: cloned images' 2024-03-23T21:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:07:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:07:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:07:39 TEST: cloned images 2024-03-23T21:07:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:07:39 TEST: cloned images 2024-03-23T21:07:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - default' 2024-03-23T21:07:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:07:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:07:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:07:39 - default 2024-03-23T21:07:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:07:39 - default 2024-03-23T21:07:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ parent_image=test_parent 2024-03-23T21:07:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ parent_snap=snap 2024-03-23T21:07:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror_parent test_parent 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror_parent/test_parent 2024-03-23T21:07:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror_parent test_parent 100 2024-03-23T21:07:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:07:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:07:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:07:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:07:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror_parent/test_parent --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:07:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:07:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:07:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2500.03 bytes/sec: 9.8 MiB/s 2024-03-23T21:07:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap 2024-03-23T21:07:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-03-23T21:07:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap 2024-03-23T21:07:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:07:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror_parent test_parent snap 2024-03-23T21:07:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-03-23T21:07:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror_parent/test_parent@snap 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image=test_clone 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-03-23T21:07:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone --image-feature layering,exclusive-lock,journaling 2024-03-23T21:07:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_clone 100 2024-03-23T21:07:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:07:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:07:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:07:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:07:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_clone --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:07:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:07:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:07:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1666.69 bytes/sec: 6.5 MiB/s 2024-03-23T21:07:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror_parent test_parent journal 2024-03-23T21:07:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=journal 2024-03-23T21:07:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror_parent/test_parent journal 2024-03-23T21:07:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2024-03-23T21:07:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror_parent/test_parent 2024-03-23T21:07:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test_parent': 2024-03-23T21:07:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:07:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:07:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 158c61352726 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.158c61352726 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling, operations 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: clone-parent 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:07:39 2024 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:07:39 2024 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:07:39 2024 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 158c61352726 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 29a3d35a-383f-4825-82f0-a6a0d4ee6d1a 2024-03-23T21:07:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror_parent test_parent 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror_parent test_parent started 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:07:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror_parent test_parent started 2024-03-23T21:07:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:07:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-03-23T21:07:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-03-23T21:07:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:07:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-03-23T21:07:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-03-23T21:07:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T21:07:45.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-03-23T21:07:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-03-23T21:07:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 4 2024-03-23T21:07:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2024-03-23T21:07:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:07:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:07:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror_parent test_parent 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:07:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror_parent test_parent 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:07:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:07:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror_parent test_parent 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_parent ']' 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror_parent/test_parent' 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror_parent/test_parent 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:07:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:07:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror_parent/test_parent 2024-03-23T21:07:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:07:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:07:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror_parent/test_parent 2024-03-23T21:07:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror_parent test_parent 2024-03-23T21:07:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror_parent 2024-03-23T21:07:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_parent 2024-03-23T21:07:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror_parent test_parent '' 2024-03-23T21:07:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror_parent 2024-03-23T21:07:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_parent 2024-03-23T21:07:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:07:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2024-03-23T21:07:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2024-03-23T21:07:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=193ad826-93ae-4fe6-82f3-baca858091ae, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2024-03-23T21:07:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=4, entry_tid=3' 2024-03-23T21:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror_parent test_parent 2024-03-23T21:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror_parent 2024-03-23T21:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_parent 2024-03-23T21:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror_parent test_parent '..*' 2024-03-23T21:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror_parent 2024-03-23T21:07:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_parent 2024-03-23T21:07:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:07:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2024-03-23T21:07:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2024-03-23T21:07:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=193ad826-93ae-4fe6-82f3-baca858091ae, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2024-03-23T21:07:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.status 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=4, entry_tid=3' 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=4, entry_tid=3' -a 'object_number=3, tag_tid=4, entry_tid=3' = 'object_number=3, tag_tid=4, entry_tid=3' 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position '' 2024-03-23T21:07:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:07:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror_parent/test_parent 2024-03-23T21:07:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_parent: 2024-03-23T21:07:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 29a3d35a-383f-4825-82f0-a6a0d4ee6d1a 2024-03-23T21:07:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:07:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2024-03-23T21:07:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:07:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:45 2024-03-23T21:07:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:47' 2024-03-23T21:07:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:07:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:07:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:07:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2024-03-23T21:07:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:07:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:45' 2024-03-23T21:07:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror_parent test_parent up+replaying '2024-03-23 21:07:45' 2024-03-23T21:07:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:45' 2024-03-23T21:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror_parent --verbose --format xml 2024-03-23T21:07:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142894302mirror.0smithi20418.2.2-954-g82316b04trueOK42934305mirror.1smithi20418.2.2-954-g82316b04falseOKtest_parent29a3d35a-383f-4825-82f0-a6a0d4ee6d1aup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}42894302mirror.0smithi2042024-03-23 21:07:45cluster275d8857e-2e9d-421a-9f31-5510b103c59dup+stoppedlocal image is primary2024-03-23 21:07:47' 2024-03-23T21:07:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/last_update' 2024-03-23T21:07:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:45' 2024-03-23T21:07:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/state' 2024-03-23T21:07:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:07:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:07:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror_parent test_parent 2024-03-23T21:07:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:07:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:07:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:07:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror_parent-test_parent.export 2024-03-23T21:07:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror_parent-test_parent.export 2024-03-23T21:07:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.export 2024-03-23T21:07:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror_parent-test_parent.export 2024-03-23T21:07:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror_parent-test_parent.export 2024-03-23T21:07:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror_parent-test_parent.export 2024-03-23T21:07:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_parent-test_parent.export 2024-03-23T21:07:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror_parent/test_parent /tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.export 2024-03-23T21:07:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:07:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror_parent/test_parent /tmp/tmp.TE9AI0RDtm/cluster1-mirror_parent-test_parent.export 2024-03-23T21:07:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:07:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_parent-test_parent.export 2024-03-23T21:07:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror_parent-test_parent.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_parent-test_parent.export 2024-03-23T21:07:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone 2024-03-23T21:07:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone started 2024-03-23T21:07:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:07:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_clone started 2024-03-23T21:07:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:07:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone 2024-03-23T21:07:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone 2024-03-23T21:07:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2024-03-23T21:07:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone 2024-03-23T21:07:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone 2024-03-23T21:07:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:07:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:07:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:07:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:07:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_clone 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_clone 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:07:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_clone 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_clone ']' 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_clone' 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_clone 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:07:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:07:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:07:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:07:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_clone 2024-03-23T21:07:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:07:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:07:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:07:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:07:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_clone 2024-03-23T21:07:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2024-03-23T21:07:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:07:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_clone 2024-03-23T21:07:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:07:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:07:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:07:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:07:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:07:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:07:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_clone 2024-03-23T21:07:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_clone 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_clone '' 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:07:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_clone.status 2024-03-23T21:07:53.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:53.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_clone.status 2024-03-23T21:07:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.status 2024-03-23T21:07:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_clone 2024-03-23T21:07:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.status 2024-03-23T21:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:07:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.status 2024-03-23T21:07:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:07:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_clone 2024-03-23T21:07:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone 2024-03-23T21:07:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_clone '..*' 2024-03-23T21:07:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone 2024-03-23T21:07:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:07:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_clone.status 2024-03-23T21:07:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_clone.status 2024-03-23T21:07:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.status 2024-03-23T21:07:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_clone 2024-03-23T21:07:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.status 2024-03-23T21:07:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:07:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:07:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:07:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:07:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:07:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.status 2024-03-23T21:07:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:07:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:07:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position 2024-03-23T21:07:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:07:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position '' 2024-03-23T21:07:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:07:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:07:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_clone 2024-03-23T21:07:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_clone: 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 22404eb6-3337-4365-a40b-32a6a71b1127 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:49 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:07:52' 2024-03-23T21:07:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:07:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:07:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:07:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:07:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:07:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:07:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:07:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:07:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:07:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:07:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:49' 2024-03-23T21:07:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_clone up+replaying '2024-03-23 21:07:49' 2024-03-23T21:07:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:07:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:07:49' 2024-03-23T21:07:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:07:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:07:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:07:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42894296mirror.0smithi2042024-03-23 21:07:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:07:52test_clone22404eb6-3337-4365-a40b-32a6a71b1127up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42934299mirror.1smithi2042024-03-23 21:07:49cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:07:52' 2024-03-23T21:07:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:07:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/last_update' 2024-03-23T21:07:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:07:49' 2024-03-23T21:07:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/state' 2024-03-23T21:07:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:07:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:07:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:07:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:07:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test_clone 2024-03-23T21:07:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:07:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test_clone.export 2024-03-23T21:07:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test_clone.export 2024-03-23T21:07:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.export 2024-03-23T21:07:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test_clone.export 2024-03-23T21:07:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test_clone.export 2024-03-23T21:07:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:07:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_clone.export 2024-03-23T21:07:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_clone.export 2024-03-23T21:07:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test_clone /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.export 2024-03-23T21:07:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:07:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test_clone /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_clone.export 2024-03-23T21:07:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:07:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_clone.export 2024-03-23T21:07:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_clone.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_clone.export 2024-03-23T21:07:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_clone 2024-03-23T21:07:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:07:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:07:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_clone 2024-03-23T21:07:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-03-23T21:07:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone 2024-03-23T21:07:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone 2024-03-23T21:07:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:07:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:07:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - clone v1' 2024-03-23T21:07:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:07:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:07:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:07:55 - clone v1 2024-03-23T21:07:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:07:55 - clone v1 2024-03-23T21:07:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster1 mirror_parent test_parent snap mirror test_clone1 2024-03-23T21:07:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-03-23T21:07:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-03-23T21:07:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap 2024-03-23T21:07:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-03-23T21:07:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone1 2024-03-23T21:07:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-03-23T21:07:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap mirror/test_clone1 --image-feature layering,exclusive-lock,journaling 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone_v1 --rbd-default-clone-format 1 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone_v1 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-03-23T21:07:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone_v1 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 1 2024-03-23T21:07:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_clone_format cluster2 mirror test_clone_v1 2024-03-23T21:07:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:07:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v1 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v1 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN { 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 1 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: $1 == "parent:" { 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: parent = $2 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: /op_features: .*clone-child/ { 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 2 2024-03-23T21:07:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:07:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: END { 2024-03-23T21:07:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: if (!parent) exit 1 2024-03-23T21:07:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: print format 2024-03-23T21:07:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: }' 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 1 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v1 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v1 started 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:07:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:07:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:07:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_clone_v1 started 2024-03-23T21:07:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:07:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:07:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-03-23T21:07:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:07:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:07:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:07:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:07:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:07:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:07:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:07:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:07:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:07:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-03-23T21:07:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:56.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-03-23T21:07:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:07:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:07:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:07:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:07:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:07:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:07:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:07:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:07:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:07:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:07:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:07:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-03-23T21:07:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:07:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:07:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:07:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:07:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:07:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:07:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:07:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:07:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:07:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:07:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2024-03-23T21:07:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-03-23T21:07:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:07:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:07:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:07:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T21:08:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:08:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2024-03-23T21:08:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:08:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:08:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:08:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2024-03-23T21:08:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:08:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:08:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:08:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:08:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:08:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:08:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:08:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_clone_format cluster1 mirror test_clone_v1 2024-03-23T21:08:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v1 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v1 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN { 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 1 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: $1 == "parent:" { 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: parent = $2 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: /op_features: .*clone-child/ { 2024-03-23T21:08:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 2 2024-03-23T21:08:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: END { 2024-03-23T21:08:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: if (!parent) exit 1 2024-03-23T21:08:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: print format 2024-03-23T21:08:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: }' 2024-03-23T21:08:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 1 2024-03-23T21:08:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_clone_v1 2024-03-23T21:08:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-03-23T21:08:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_clone_v1 2024-03-23T21:08:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-03-23T21:08:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v1 2024-03-23T21:08:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v1 2024-03-23T21:08:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:08:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster1 mirror test_clone1 2024-03-23T21:08:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone1 2024-03-23T21:08:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster1 mirror test_clone1 2024-03-23T21:08:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone1 2024-03-23T21:08:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone1 2024-03-23T21:08:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone1 2024-03-23T21:08:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:08:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot_retry cluster2 mirror_parent test_parent snap 2024-03-23T21:08:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-03-23T21:08:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror_parent test_parent snap 2024-03-23T21:08:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-03-23T21:08:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror_parent/test_parent@snap 2024-03-23T21:08:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap 2024-03-23T21:08:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-03-23T21:08:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap 2024-03-23T21:08:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-03-23T21:08:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - clone v2' 2024-03-23T21:08:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:08:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:08:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:08:01 - clone v2 2024-03-23T21:08:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:08:01 - clone v2 2024-03-23T21:08:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ parent_snap=snap_v2 2024-03-23T21:08:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_v2 2024-03-23T21:08:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2024-03-23T21:08:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2024-03-23T21:08:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-03-23T21:08:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-03-23T21:08:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap_v2 2024-03-23T21:08:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-03-23T21:08:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone_v2 2024-03-23T21:08:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-03-23T21:08:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_clone_format cluster2 mirror test_clone_v2 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v2 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v2 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN { 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 1 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: $1 == "parent:" { 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: parent = $2 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: /op_features: .*clone-child/ { 2024-03-23T21:08:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 2 2024-03-23T21:08:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: END { 2024-03-23T21:08:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: if (!parent) exit 1 2024-03-23T21:08:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: print format 2024-03-23T21:08:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: }' 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v2 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v2 started 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_clone_v2 started 2024-03-23T21:08:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:08:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:08:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:08:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:08:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:08:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:08:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:08:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:08:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:03.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:08:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:08:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:08:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-03-23T21:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:08:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-03-23T21:08:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:08:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:08:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:08:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:08:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-03-23T21:08:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:08:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:08:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-03-23T21:08:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T21:08:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:08:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-03-23T21:08:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:08:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:08:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:08:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-03-23T21:08:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 4 2024-03-23T21:08:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:08:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:08:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:08:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:08:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-03-23T21:08:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:08:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:08:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:08:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:08:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2024-03-23T21:08:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:08:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:08:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:08:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:08:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:08:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:08:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:08:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_clone_format cluster1 mirror test_clone_v2 2024-03-23T21:08:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v2 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN { 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 1 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: $1 == "parent:" { 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: parent = $2 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: /op_features: .*clone-child/ { 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 2 2024-03-23T21:08:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-03-23T21:08:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: END { 2024-03-23T21:08:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: if (!parent) exit 1 2024-03-23T21:08:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: print format 2024-03-23T21:08:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: }' 2024-03-23T21:08:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v2 2024-03-23T21:08:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 2024-03-23T21:08:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_v2 2024-03-23T21:08:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2024-03-23T21:08:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-03-23T21:08:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2024-03-23T21:08:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-03-23T21:08:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 8 42599bb4-b22e-4375-ab38-afab3f95688b 128 MiB Sat Mar 23 21:08:02 2024 trash (snap_v2) 2024-03-23T21:08:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-03-23T21:08:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-03-23T21:08:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-03-23T21:08:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:08:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-03-23T21:08:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-03-23T21:08:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 8 e42e660c-4e59-415b-9ad3-f4af324eba91 128 MiB Sat Mar 23 21:08:05 2024 trash (snap_v2) 2024-03-23T21:08:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_clone_v2 2024-03-23T21:08:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_clone_v2 2024-03-23T21:08:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v2 2024-03-23T21:08:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v2 2024-03-23T21:08:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:08:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test_clone_v2 deleted 2024-03-23T21:08:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:08:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test_clone_v2 2024-03-23T21:08:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v2 2024-03-23T21:08:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2 2024-03-23T21:08:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2024-03-23T21:08:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:08:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test_clone_v2 deleted '' 2024-03-23T21:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:08:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test_clone_v2 2024-03-23T21:08:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v2 2024-03-23T21:08:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2 2024-03-23T21:08:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2024-03-23T21:08:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:08:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:08:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2024-03-23T21:08:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-03-23T21:08:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-03-23T21:08:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-03-23T21:08:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - clone v2 non-primary' 2024-03-23T21:08:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:08:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:08:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:08:16 - clone v2 non-primary 2024-03-23T21:08:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:08:16 - clone v2 non-primary 2024-03-23T21:08:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_v2 2024-03-23T21:08:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2024-03-23T21:08:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2024-03-23T21:08:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-03-23T21:08:18.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-03-23T21:08:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:08:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2024-03-23T21:08:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test_parent': 2024-03-23T21:08:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:08:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:08:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2024-03-23T21:08:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10ced39d1586 2024-03-23T21:08:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10ced39d1586 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:07:42 2024 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:07:42 2024 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:07:42 2024 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10ced39d1586 2024-03-23T21:08:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:08:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:08:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 29a3d35a-383f-4825-82f0-a6a0d4ee6d1a 2024-03-23T21:08:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap_v2 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone_v2 2024-03-23T21:08:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-03-23T21:08:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2024-03-23T21:08:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_v2 2024-03-23T21:08:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2024-03-23T21:08:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-03-23T21:08:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2024-03-23T21:08:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2024-03-23T21:08:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-03-23T21:08:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-03-23T21:08:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-03-23T21:08:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 10 2cd86aaa-b80c-4c57-a18c-82180202c7dd 128 MiB Sat Mar 23 21:08:19 2024 trash (snap_v2) 2024-03-23T21:08:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster1 mirror test_clone_v2 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster1 mirror test_clone_v2 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-03-23T21:08:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone_v2 2024-03-23T21:08:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone_v2 2024-03-23T21:08:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:08:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:23.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-03-23T21:08:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-03-23T21:08:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-03-23T21:08:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-03-23T21:08:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror_parent test_parent 2024-03-23T21:08:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror_parent test_parent 2024-03-23T21:08:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-03-23T21:08:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-03-23T21:08:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror_parent/test_parent 2024-03-23T21:08:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror_parent/test_parent 2024-03-23T21:08:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:08:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: data pool' 2024-03-23T21:08:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:08:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:08:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:08:24 TEST: data pool 2024-03-23T21:08:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:08:24 TEST: data pool 2024-03-23T21:08:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ dp_image=test_data_pool 2024-03-23T21:08:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test_data_pool 128 --data-pool mirror_parent 2024-03-23T21:08:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:08:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-03-23T21:08:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-03-23T21:08:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/test_data_pool 2024-03-23T21:08:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_data_pool cluster2 mirror test_data_pool 2024-03-23T21:08:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 info mirror/test_data_pool 2024-03-23T21:08:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk '$1 == "data_pool:" {print $2}' 2024-03-23T21:08:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ data_pool=mirror_parent 2024-03-23T21:08:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test mirror_parent = mirror_parent 2024-03-23T21:08:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_data_pool 2024-03-23T21:08:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_data_pool started 2024-03-23T21:08:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:08:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_data_pool started 2024-03-23T21:08:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:08:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:08:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_data_pool 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:08:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:08:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2024-03-23T21:08:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:08:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:08:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:08:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2024-03-23T21:08:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-03-23T21:08:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2024-03-23T21:08:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:08:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:08:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2024-03-23T21:08:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-03-23T21:08:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:08:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:08:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:08:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:08:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2024-03-23T21:08:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-03-23T21:08:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:08:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:08:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2024-03-23T21:08:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-03-23T21:08:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T21:08:29.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:08:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2024-03-23T21:08:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:08:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:08:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2024-03-23T21:08:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-03-23T21:08:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:08:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2024-03-23T21:08:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:08:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2024-03-23T21:08:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:08:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:08:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:08:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:08:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:08:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:08:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:08:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_data_pool cluster1 mirror test_data_pool 2024-03-23T21:08:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 info mirror/test_data_pool 2024-03-23T21:08:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk '$1 == "data_pool:" {print $2}' 2024-03-23T21:08:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ data_pool=mirror_parent 2024-03-23T21:08:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test mirror_parent = mirror_parent 2024-03-23T21:08:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test_data_pool snap1 2024-03-23T21:08:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap1 2024-03-23T21:08:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_data_pool 100 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.36 bytes/sec: 8.1 MiB/s 2024-03-23T21:08:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test_data_pool snap2 2024-03-23T21:08:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap2 2024-03-23T21:08:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_data_pool 100 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1923.1 bytes/sec: 7.5 MiB/s 2024-03-23T21:08:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_data_pool 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_data_pool 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:08:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:08:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:08:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:08:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:08:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_data_pool 2024-03-23T21:08:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:08:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-03-23T21:08:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2024-03-23T21:08:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:08:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:08:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:08:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:08:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:08:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:08:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-03-23T21:08:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:08:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:08:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:08:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2024-03-23T21:08:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2024-03-23T21:08:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:08:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-03-23T21:08:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:08:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:08:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:08:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T21:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2024-03-23T21:08:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2024-03-23T21:08:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2024-03-23T21:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-03-23T21:08:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:08:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:08:31.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:08:31.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T21:08:31.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2024-03-23T21:08:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=101' 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=3, entry_tid=101' 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' '!=' '' 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' = '' 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=3, entry_tid=101' 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:08:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:08:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=5 2024-03-23T21:08:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=101' 2024-03-23T21:08:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:08:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:08:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=3 2024-03-23T21:08:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:08:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:08:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T21:08:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=101' 2024-03-23T21:08:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:08:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:08:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=101 2024-03-23T21:08:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 5 = 3 -a 101 -le 101 2024-03-23T21:08:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:08:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:08:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:08:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_data_pool 2024-03-23T21:08:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:08:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-03-23T21:08:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2024-03-23T21:08:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:08:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:08:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:08:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:08:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-03-23T21:08:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:08:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:08:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:08:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:08:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:08:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-03-23T21:08:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:08:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:08:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:08:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T21:08:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=13], [object_number=0, tag_tid=5, entry_tid=12], [object_number=3, tag_tid=5, entry_tid=11], [object_number=2, tag_tid=5, entry_tid=10]]], state=connected] 2024-03-23T21:08:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2024-03-23T21:08:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2024-03-23T21:08:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:08:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-03-23T21:08:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T21:08:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=13], [object_number=0, tag_tid=5, entry_tid=12], [object_number=3, tag_tid=5, entry_tid=11], [object_number=2, tag_tid=5, entry_tid=10]]], state=connected] 2024-03-23T21:08:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=13' 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=13' 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=5, entry_tid=13' '!=' 'object_number=1, tag_tid=3, entry_tid=101' 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=5, entry_tid=13' = 'object_number=1, tag_tid=3, entry_tid=101' 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=5, entry_tid=13' 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:08:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=5 2024-03-23T21:08:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=13' 2024-03-23T21:08:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:08:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:08:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=5 2024-03-23T21:08:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:08:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:08:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T21:08:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=13' 2024-03-23T21:08:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:08:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:08:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=13 2024-03-23T21:08:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 5 = 5 -a 101 -le 13 2024-03-23T21:08:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:08:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:08:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:08:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_data_pool 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:08:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:08:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:08:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:08:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:08:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:08:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:08:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:08:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:08:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-03-23T21:08:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:08:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:08:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:08:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:08:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:08:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:08:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_data_pool 2024-03-23T21:08:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:08:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-03-23T21:08:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:08:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:08:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:08:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T21:08:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T21:08:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-03-23T21:08:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T21:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-03-23T21:08:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.status 2024-03-23T21:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=101' 2024-03-23T21:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position 2024-03-23T21:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:08:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position '' 2024-03-23T21:08:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:08:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:08:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:08:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:08:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:08:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_data_pool 2024-03-23T21:08:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_data_pool: 2024-03-23T21:08:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a305d156-5d82-4dd0-9417-0eb0889c47c2 2024-03-23T21:08:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:08:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:08:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:08:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:08:28 2024-03-23T21:08:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:08:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:08:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:08:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:08:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:08:32' 2024-03-23T21:08:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:08:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:08:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:08:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:08:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:08:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:08:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:08:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:08:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:08:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:08:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:08:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:08:28' 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_data_pool up+replaying '2024-03-23 21:08:28' 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:08:28' 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:08:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:08:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK242894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42894296mirror.0smithi2042024-03-23 21:08:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:08:32test_data_poola305d156-5d82-4dd0-9417-0eb0889c47c2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42934299mirror.1smithi2042024-03-23 21:08:28cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:08:32' 2024-03-23T21:08:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:08:34.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update' 2024-03-23T21:08:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:08:28' 2024-03-23T21:08:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state' 2024-03-23T21:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:08:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:08:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:08:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test_data_pool@snap1 2024-03-23T21:08:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool@snap1 2024-03-23T21:08:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:08:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap1 /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:08:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap1 /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:08:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap1.export 2024-03-23T21:08:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test_data_pool@snap2 2024-03-23T21:08:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool@snap2 2024-03-23T21:08:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:08:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap2 /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap2 /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap2.export 2024-03-23T21:08:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:08:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap2.export 2024-03-23T21:08:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool@snap2.export 2024-03-23T21:08:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test_data_pool 2024-03-23T21:08:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:08:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test_data_pool.export 2024-03-23T21:08:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test_data_pool.export 2024-03-23T21:08:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.export 2024-03-23T21:08:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test_data_pool.export 2024-03-23T21:08:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test_data_pool.export 2024-03-23T21:08:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:08:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool.export 2024-03-23T21:08:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool.export 2024-03-23T21:08:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.export 2024-03-23T21:08:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:08:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool.export 2024-03-23T21:08:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:08:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool.export 2024-03-23T21:08:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_data_pool.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test_data_pool.export 2024-03-23T21:08:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_data_pool 2024-03-23T21:08:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_data_pool 2024-03-23T21:08:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:08:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_data_pool 2024-03-23T21:08:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-03-23T21:08:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_data_pool 2024-03-23T21:08:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:08:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: disable mirroring / delete non-primary image' 2024-03-23T21:08:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:08:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:08:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:08:37 TEST: disable mirroring / delete non-primary image 2024-03-23T21:08:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:08:37 TEST: disable mirroring / delete non-primary image 2024-03-23T21:08:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image2=test2 2024-03-23T21:08:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image3=test3 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image4=test4 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image5=test5 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test2 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:08:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test2 2024-03-23T21:08:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test2 100 2024-03-23T21:08:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2500.03 bytes/sec: 9.8 MiB/s 2024-03-23T21:08:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test2 snap1 2024-03-23T21:08:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap1 2024-03-23T21:08:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test2 snap2 2024-03-23T21:08:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap2 2024-03-23T21:08:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test2 = test4 ']' 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test2 = test5 ']' 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test2 100 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.36 bytes/sec: 8.1 MiB/s 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test2 present 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:08:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:08:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-03-23T21:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:08:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-03-23T21:08:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:08:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c8d2ad50a6 2024-03-23T21:08:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c8d2ad50a6 2024-03-23T21:08:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c8d2ad50a6 2024-03-23T21:08:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:08:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:08:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2024-03-23T21:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-03-23T21:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2024-03-23T21:08:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-03-23T21:08:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-03-23T21:08:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:08:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2024-03-23T21:08:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test2': 2024-03-23T21:08:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:08:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:08:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 3 2024-03-23T21:08:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c8d2ad50a6 2024-03-23T21:08:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c8d2ad50a6 2024-03-23T21:08:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:08:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:08:39 2024 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:08:39 2024 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:08:39 2024 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c8d2ad50a6 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: da8f5640-750e-4925-b442-799caaa61d2b 2024-03-23T21:08:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-03-23T21:08:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-03-23T21:08:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test3 2024-03-23T21:08:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:08:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:08:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test3 2024-03-23T21:08:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test3 100 2024-03-23T21:08:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.36 bytes/sec: 8.1 MiB/s 2024-03-23T21:08:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test3 snap1 2024-03-23T21:08:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap1 2024-03-23T21:08:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test3 snap2 2024-03-23T21:08:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap2 2024-03-23T21:08:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:44.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test3 = test4 ']' 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test3 = test5 ']' 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test3 100 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.36 bytes/sec: 8.1 MiB/s 2024-03-23T21:08:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test3 present 2024-03-23T21:08:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-03-23T21:08:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test3: (2) No such file or directory 2024-03-23T21:08:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:08:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:08:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test3 present '' 2024-03-23T21:08:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:08:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-03-23T21:08:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-03-23T21:08:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-03-23T21:08:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test3: (2) No such file or directory 2024-03-23T21:08:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:08:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:08:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test3 present '' 2024-03-23T21:08:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:08:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-03-23T21:08:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-03-23T21:08:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-03-23T21:08:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10cb60b234e 2024-03-23T21:08:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10cb60b234e 2024-03-23T21:08:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10cb60b234e 2024-03-23T21:08:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test3 snap2 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2 2024-03-23T21:08:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test3': 2024-03-23T21:08:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10cb60b234e 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10cb60b234e 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:08:44 2024 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:08:44 2024 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:08:44 2024 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10cb60b234e 2024-03-23T21:08:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:08:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:08:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 85a35c46-0747-4aaa-8eea-a6b826bd445c 2024-03-23T21:08:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-03-23T21:08:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-03-23T21:08:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test4 2024-03-23T21:08:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:08:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:08:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test4 2024-03-23T21:08:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test4 100 2024-03-23T21:08:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.36 bytes/sec: 8.1 MiB/s 2024-03-23T21:08:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test4 snap1 2024-03-23T21:08:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap1 2024-03-23T21:08:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test4 snap2 2024-03-23T21:08:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap2 2024-03-23T21:08:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test4 = test4 ']' 2024-03-23T21:08:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror test4 snap1 2024-03-23T21:08:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap1 2024-03-23T21:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror test4 snap2 2024-03-23T21:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap2 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test4 100 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1923.1 bytes/sec: 7.5 MiB/s 2024-03-23T21:08:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test4 present 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:08:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-03-23T21:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:08:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:08:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:08:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-03-23T21:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-03-23T21:08:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:08:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:08:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:08:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:08:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10cb3849aac0 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10cb3849aac0 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10cb3849aac0 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test4': 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10cb3849aac0 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10cb3849aac0 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:08:48 2024 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:08:48 2024 2024-03-23T21:08:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:08:48 2024 2024-03-23T21:08:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-03-23T21:08:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10cb3849aac0 2024-03-23T21:08:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:08:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:08:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: ec333e51-baed-4c2e-9038-4cf57fd06fd9 2024-03-23T21:08:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-03-23T21:08:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-03-23T21:08:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test5 2024-03-23T21:08:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:08:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:08:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:08:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test5 2024-03-23T21:08:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test5 100 2024-03-23T21:08:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1785.74 bytes/sec: 7.0 MiB/s 2024-03-23T21:08:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test5 snap1 2024-03-23T21:08:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap1 2024-03-23T21:08:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test5 snap2 2024-03-23T21:08:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap2 2024-03-23T21:08:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:08:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test5 = test4 ']' 2024-03-23T21:08:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test5 = test5 ']' 2024-03-23T21:08:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror test5 snap1 2024-03-23T21:08:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap1 2024-03-23T21:08:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror test5 snap2 2024-03-23T21:08:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap2 2024-03-23T21:08:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test5 100 2024-03-23T21:08:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:08:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:08:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:08:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:08:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:08:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:08:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1785.74 bytes/sec: 7.0 MiB/s 2024-03-23T21:08:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test5 present 2024-03-23T21:08:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-03-23T21:08:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test5: (2) No such file or directory 2024-03-23T21:08:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:08:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:08:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test5 present '' 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-03-23T21:08:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-03-23T21:08:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test5: (2) No such file or directory 2024-03-23T21:08:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:08:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:08:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test5 present '' 2024-03-23T21:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-03-23T21:08:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-03-23T21:08:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-03-23T21:08:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c85fd665d6 2024-03-23T21:08:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c85fd665d6 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c85fd665d6 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test5 snap2 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:08:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:08:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test5@snap2 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test5': 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c85fd665d6 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c85fd665d6 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:08:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:08:52 2024 2024-03-23T21:08:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:08:52 2024 2024-03-23T21:08:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:08:52 2024 2024-03-23T21:08:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-03-23T21:08:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c85fd665d6 2024-03-23T21:08:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:08:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:08:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 99982d0f-3221-42a8-ad8d-3c5ada58627b 2024-03-23T21:08:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-03-23T21:08:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-03-23T21:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=image 2024-03-23T21:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-03-23T21:08:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:note: changing mirroring mode from pool to image 2024-03-23T21:08:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-03-23T21:08:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror test2 2024-03-23T21:08:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test2 2024-03-23T21:08:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2024-03-23T21:08:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-03-23T21:08:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror test4 2024-03-23T21:08:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:08:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test4 2024-03-23T21:08:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2024-03-23T21:08:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror test5 snap1 2024-03-23T21:08:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap1 2024-03-23T21:08:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror test5 snap2 2024-03-23T21:08:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap2 2024-03-23T21:08:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image3} ${image5} 2024-03-23T21:08:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test3 snap1 2024-03-23T21:08:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap1 2024-03-23T21:08:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-03-23T21:08:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test3 snap2 2024-03-23T21:08:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap2 2024-03-23T21:08:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-03-23T21:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test3 2024-03-23T21:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test3 2024-03-23T21:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:08:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test3 2024-03-23T21:08:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test3 2024-03-23T21:08:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:08:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image3} ${image5} 2024-03-23T21:08:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test5 snap1 2024-03-23T21:08:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:08:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap1 2024-03-23T21:08:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-03-23T21:08:58.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test5 snap2 2024-03-23T21:08:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:08:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap2 2024-03-23T21:08:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-03-23T21:08:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test5 2024-03-23T21:08:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:08:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:08:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test5 2024-03-23T21:08:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:08:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:08:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test5 2024-03-23T21:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test5 2024-03-23T21:08:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:08:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:08:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-03-23T21:08:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test2 deleted 2024-03-23T21:08:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:08:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:08:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:08:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:08:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:08:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:08:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:08:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10c8d2ad50a6 2024-03-23T21:08:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:08:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 deleted 10c8d2ad50a6 2024-03-23T21:08:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:08:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:08:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:08:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:08:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c8d2ad50a6 2024-03-23T21:08:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:08:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:08:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:08:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:08:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:08:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:08:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:08:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:08:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c8d2ad50a6 2024-03-23T21:08:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c8d2ad50a6 2024-03-23T21:08:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c8d2ad50a6 -o 10c8d2ad50a6 = 10c8d2ad50a6 2024-03-23T21:08:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:08:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:08:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:08:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 deleted 10c8d2ad50a6 2024-03-23T21:09:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:09:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c8d2ad50a6 2024-03-23T21:09:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:09:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:09:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-03-23T21:09:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test3 deleted 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:09:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-03-23T21:09:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test3: (2) No such file or directory 2024-03-23T21:09:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:09:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test3 deleted '' 2024-03-23T21:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-03-23T21:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-03-23T21:09:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-03-23T21:09:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-03-23T21:09:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test3: (2) No such file or directory 2024-03-23T21:09:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:09:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test4 deleted 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:09:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-03-23T21:09:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:09:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:01.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:09:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 deleted '' 2024-03-23T21:09:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:09:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:09:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:09:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-03-23T21:09:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:09:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test5 deleted 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-03-23T21:09:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:01.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test5: (2) No such file or directory 2024-03-23T21:09:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:09:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:09:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test5 deleted '' 2024-03-23T21:09:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-03-23T21:09:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:09:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-03-23T21:09:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-03-23T21:09:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-03-23T21:09:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test5: (2) No such file or directory 2024-03-23T21:09:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:09:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror pool 2024-03-23T21:09:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=pool 2024-03-23T21:09:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool 2024-03-23T21:09:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:note: changing mirroring mode from image to pool 2024-03-23T21:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-03-23T21:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_journaling cluster2 mirror test2 2024-03-23T21:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 feature enable mirror/test2 journaling 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test2 present 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:09:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:09:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-03-23T21:09:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:09:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:09:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-03-23T21:09:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:09:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:09:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:09:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:01.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-03-23T21:09:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:09:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-03-23T21:09:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:09:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:09:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:09:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:02.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-03-23T21:09:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:09:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:09:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-03-23T21:09:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:09:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:09:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:09:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-03-23T21:09:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:09:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:09:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-03-23T21:09:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:09:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-03-23T21:09:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-03-23T21:09:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10cb14c121d8 2024-03-23T21:09:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10cb14c121d8 2024-03-23T21:09:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10cb14c121d8 2024-03-23T21:09:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:09:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:09:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2024-03-23T21:09:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-03-23T21:09:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:09:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2024-03-23T21:09:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test2': 2024-03-23T21:09:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:09:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:09:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-03-23T21:09:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10cb14c121d8 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10cb14c121d8 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:09:08 2024 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:09:08 2024 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:09:08 2024 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10cb14c121d8 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a531096b-315e-4e71-9418-5786990aade9 2024-03-23T21:09:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test2 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test2 started 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:09:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test2 started 2024-03-23T21:09:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test2 2024-03-23T21:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:09:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:09:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:09:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test2 2024-03-23T21:09:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:09:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:09:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:09:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:09:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:09:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test2 2024-03-23T21:09:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test2'] 2024-03-23T21:09:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test2 2024-03-23T21:09:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:09:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:09:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:09:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:09:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:09:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:09:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test2 2024-03-23T21:09:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:09:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:09:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:09:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:09:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:09:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:09:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:09:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test2 2024-03-23T21:09:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:09:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test2 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:09:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:09:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test2 2024-03-23T21:09:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test2 ']' 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test2' 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:09:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2 2024-03-23T21:09:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test2'] 2024-03-23T21:09:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test2 2024-03-23T21:09:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:09:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:09:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test2 2024-03-23T21:09:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test2 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test2 '' 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test2.status 2024-03-23T21:09:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test2.status 2024-03-23T21:09:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2024-03-23T21:09:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test2 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test2 '..*' 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:09:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test2.status 2024-03-23T21:09:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test2.status 2024-03-23T21:09:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2024-03-23T21:09:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' '' 2024-03-23T21:09:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:09:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = '' 2024-03-23T21:09:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:09:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=1 2024-03-23T21:09:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:09:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-03-23T21:09:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:09:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-03-23T21:09:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:09:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-03-23T21:09:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 1 -a 3 -le 1 2024-03-23T21:09:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:09:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:09:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test2 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test2 ']' 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test2' 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2 2024-03-23T21:09:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:09:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:09:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2 2024-03-23T21:09:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test2'] 2024-03-23T21:09:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test2 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:09:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test2 2024-03-23T21:09:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test2 2024-03-23T21:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test2 '' 2024-03-23T21:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:09:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test2.status 2024-03-23T21:09:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test2.status 2024-03-23T21:09:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2024-03-23T21:09:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test2 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test2 '..*' 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test2.status 2024-03-23T21:09:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test2.status 2024-03-23T21:09:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2024-03-23T21:09:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.status 2024-03-23T21:09:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test2 2024-03-23T21:09:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:09:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:09:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test2.export 2024-03-23T21:09:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test2.export 2024-03-23T21:09:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.export 2024-03-23T21:09:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test2.export 2024-03-23T21:09:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test2.export 2024-03-23T21:09:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test2.export 2024-03-23T21:09:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test2.export 2024-03-23T21:09:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test2 /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.export 2024-03-23T21:09:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:09:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test2 /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test2.export 2024-03-23T21:09:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:09:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test2.export 2024-03-23T21:09:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test2.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test2.export 2024-03-23T21:09:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-03-23T21:09:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_journaling cluster2 mirror test4 2024-03-23T21:09:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 feature enable mirror/test4 journaling 2024-03-23T21:09:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test4 present 2024-03-23T21:09:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:09:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:09:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:09:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-03-23T21:09:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:09:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:09:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-03-23T21:09:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:09:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:09:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:09:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-03-23T21:09:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:09:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-03-23T21:09:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:09:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:09:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:09:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-03-23T21:09:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:09:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:09:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-03-23T21:09:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:09:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-03-23T21:09:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-03-23T21:09:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c8e9a1a45d 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c8e9a1a45d 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c8e9a1a45d 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-03-23T21:09:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:09:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2024-03-23T21:09:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test4': 2024-03-23T21:09:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c8e9a1a45d 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c8e9a1a45d 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:09:15 2024 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:09:15 2024 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:09:15 2024 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c8e9a1a45d 2024-03-23T21:09:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:09:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:09:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 360e8a7c-d090-48c6-919a-1a1fd81371bb 2024-03-23T21:09:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-03-23T21:09:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test4 2024-03-23T21:09:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test4 started 2024-03-23T21:09:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:09:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test4 started 2024-03-23T21:09:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:09:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:09:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:09:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test4 2024-03-23T21:09:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:09:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:09:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:09:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:09:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:09:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:09:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test4 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:09:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:09:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:09:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:09:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:09:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test4 2024-03-23T21:09:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test4'] 2024-03-23T21:09:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test4 2024-03-23T21:09:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:09:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:09:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:09:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:09:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:09:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:09:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test4 2024-03-23T21:09:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test4'] 2024-03-23T21:09:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:09:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:09:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test4 2024-03-23T21:09:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:09:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:09:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:09:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:09:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test4 2024-03-23T21:09:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:09:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:09:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:09:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:09:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:09:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:09:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:09:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test4 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test4 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:09:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:09:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test4 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test4 ']' 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test4' 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test4 2024-03-23T21:09:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:09:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:09:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test4 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test4 2024-03-23T21:09:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test4 2024-03-23T21:09:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test4 '' 2024-03-23T21:09:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:09:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test4.status 2024-03-23T21:09:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test4.status 2024-03-23T21:09:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4 2024-03-23T21:09:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test4 2024-03-23T21:09:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test4 '..*' 2024-03-23T21:09:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:09:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test4.status 2024-03-23T21:09:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test4.status 2024-03-23T21:09:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4 2024-03-23T21:09:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' '' 2024-03-23T21:09:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:09:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = '' 2024-03-23T21:09:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:09:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=1 2024-03-23T21:09:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:09:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-03-23T21:09:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:09:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-03-23T21:09:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2024-03-23T21:09:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:09:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-03-23T21:09:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 1 -a 3 -le 1 2024-03-23T21:09:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:09:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:09:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:09:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test4 2024-03-23T21:09:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:09:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test4 ']' 2024-03-23T21:09:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test4' 2024-03-23T21:09:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test4 2024-03-23T21:09:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:09:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:09:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test4 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test4 2024-03-23T21:09:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test4 2024-03-23T21:09:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test4 '' 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test4.status 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test4.status 2024-03-23T21:09:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4 2024-03-23T21:09:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test4 2024-03-23T21:09:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test4 '..*' 2024-03-23T21:09:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-03-23T21:09:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:09:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test4.status 2024-03-23T21:09:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test4.status 2024-03-23T21:09:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4 2024-03-23T21:09:21.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:09:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.status 2024-03-23T21:09:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:09:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test4 2024-03-23T21:09:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:09:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:09:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test4.export 2024-03-23T21:09:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test4.export 2024-03-23T21:09:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.export 2024-03-23T21:09:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test4.export 2024-03-23T21:09:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test4.export 2024-03-23T21:09:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test4.export 2024-03-23T21:09:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test4.export 2024-03-23T21:09:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test4 /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.export 2024-03-23T21:09:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:09:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test4 /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test4.export 2024-03-23T21:09:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:09:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test4.export 2024-03-23T21:09:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test4.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test4.export 2024-03-23T21:09:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: remove mirroring pool' 2024-03-23T21:09:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:09:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:09:21.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:09:21 TEST: remove mirroring pool 2024-03-23T21:09:21.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:09:21 TEST: remove mirroring pool 2024-03-23T21:09:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pool=pool_to_remove 2024-03-23T21:09:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-03-23T21:09:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:09:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool create pool_to_remove 16 16 2024-03-23T21:09:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'pool_to_remove' already exists 2024-03-23T21:09:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:09:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 pool init pool_to_remove 2024-03-23T21:09:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable pool_to_remove pool 2024-03-23T21:09:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-03-23T21:09:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:09:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool create pool_to_remove 16 16 2024-03-23T21:09:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'pool_to_remove' already exists 2024-03-23T21:09:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:09:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 pool init pool_to_remove 2024-03-23T21:09:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable pool_to_remove pool 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster1 pool_to_remove cluster2 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster2 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:09:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-03-23T21:09:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-03-23T21:09:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:09:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:09:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool peer add pool_to_remove cluster2 2024-03-23T21:09:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=c2cf3bfb-cfa2-4005-8f37-300af120a970 2024-03-23T21:09:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T21:09:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:09:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-03-23T21:09:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-03-23T21:09:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:09:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster2 pool_to_remove cluster1 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster1 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-03-23T21:09:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-03-23T21:09:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:09:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:09:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool peer add pool_to_remove cluster1 2024-03-23T21:09:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=ce616c71-c8bf-486a-8708-c4a39fc331b4 2024-03-23T21:09:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T21:09:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rdp_image=test_remove_data_pool 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 pool_to_remove test 128 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:09:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-03-23T21:09:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-03-23T21:09:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling pool_to_remove/test 2024-03-23T21:09:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test_remove_data_pool 128 --data-pool pool_to_remove 2024-03-23T21:09:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool pool_to_remove mirror/test_remove_data_pool 2024-03-23T21:09:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 pool_to_remove test 100 2024-03-23T21:09:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:09:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:09:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:09:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench pool_to_remove/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:09:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:09:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:09:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2272.75 bytes/sec: 8.9 MiB/s 2024-03-23T21:09:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_remove_data_pool 100 2024-03-23T21:09:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:09:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:09:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:09:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_remove_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:09:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:09:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:09:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2777.81 bytes/sec: 11 MiB/s 2024-03-23T21:09:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 pool_to_remove test 2024-03-23T21:09:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:09:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 pool_to_remove test 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:09:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:09:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 pool_to_remove test 2024-03-23T21:09:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:09:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:09:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush pool_to_remove/test' 2024-03-23T21:09:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:09:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:09:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:09:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:09:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:09:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:09:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:09:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:09:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:09:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:09:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:09:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:09:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:09:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:09:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:09:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 pool_to_remove test 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 pool_to_remove test '' 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:09:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2024-03-23T21:09:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=93acd38d-e817-4e6c-bf50-6f611850127a, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=101], [object_number=0, tag_tid=1, entry_tid=100], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-03-23T21:09:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 pool_to_remove test 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 pool_to_remove test '..*' 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2024-03-23T21:09:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=93acd38d-e817-4e6c-bf50-6f611850127a, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=101], [object_number=0, tag_tid=1, entry_tid=100], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-03-23T21:09:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=101' 2024-03-23T21:09:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=1, tag_tid=1, entry_tid=101' 2024-03-23T21:09:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=101' '!=' '' 2024-03-23T21:09:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:09:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=101' = '' 2024-03-23T21:09:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=101' 2024-03-23T21:09:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:09:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-03-23T21:09:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=101' 2024-03-23T21:09:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:09:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-03-23T21:09:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:09:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-03-23T21:09:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=101' 2024-03-23T21:09:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:09:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=101 2024-03-23T21:09:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 1 -a 3 -le 101 2024-03-23T21:09:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:09:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:09:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:09:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 pool_to_remove test 2024-03-23T21:09:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:09:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush pool_to_remove/test' 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 pool_to_remove test 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 pool_to_remove test '' 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:09:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2024-03-23T21:09:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:38.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=93acd38d-e817-4e6c-bf50-6f611850127a, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-03-23T21:09:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:38.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 pool_to_remove test 2024-03-23T21:09:38.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:38.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 pool_to_remove test '..*' 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2024-03-23T21:09:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=93acd38d-e817-4e6c-bf50-6f611850127a, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-03-23T21:09:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-03-23T21:09:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-03-23T21:09:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=1, entry_tid=101' 2024-03-23T21:09:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:09:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=101' 2024-03-23T21:09:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-03-23T21:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-03-23T21:09:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-03-23T21:09:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:09:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=2 2024-03-23T21:09:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:09:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-03-23T21:09:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-03-23T21:09:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:09:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:09:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-03-23T21:09:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 -a 3 -le 1 2024-03-23T21:09:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:09:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:09:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:09:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 pool_to_remove test 2024-03-23T21:09:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush pool_to_remove/test' 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-03-23T21:09:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 pool_to_remove test 2024-03-23T21:09:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 pool_to_remove test '' 2024-03-23T21:09:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:09:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2024-03-23T21:09:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=93acd38d-e817-4e6c-bf50-6f611850127a, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 pool_to_remove test 2024-03-23T21:09:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 pool_to_remove test '..*' 2024-03-23T21:09:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-03-23T21:09:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:09:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:09:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2024-03-23T21:09:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=93acd38d-e817-4e6c-bf50-6f611850127a, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-pool_to_remove-test.status 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position '' 2024-03-23T21:09:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:09:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:09:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:09:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:09:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:09:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status pool_to_remove/test 2024-03-23T21:09:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:09:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 077c1d67-af21-4b8b-8465-ddc4355a838e 2024-03-23T21:09:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:09:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:09:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:09:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:09:37 2024-03-23T21:09:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:09:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:09:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:09:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:09:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:09:38' 2024-03-23T21:09:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:09:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:09:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:09:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:09:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:09:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:09:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:09:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:09:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:09:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:09:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:09:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:09:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:09:37' 2024-03-23T21:09:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 pool_to_remove test up+replaying '2024-03-23 21:09:37' 2024-03-23T21:09:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-03-23T21:09:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:09:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:09:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:09:37' 2024-03-23T21:09:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:09:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:09:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status pool_to_remove --verbose --format xml 2024-03-23T21:09:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142895748mirror.0smithi20418.2.2-954-g82316b04falseOK42935751mirror.1smithi20418.2.2-954-g82316b04trueOKtest077c1d67-af21-4b8b-8465-ddc4355a838eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42895748mirror.0smithi2042024-03-23 21:09:37cluster2e76298f9-d378-4246-88f5-47b18fbfd038up+stoppedlocal image is primary2024-03-23 21:09:38' 2024-03-23T21:09:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:09:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:09:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:09:37' 2024-03-23T21:09:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:09:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:09:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:09:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:09:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:09:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_remove_data_pool 2024-03-23T21:09:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:09:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_remove_data_pool 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:09:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:09:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_remove_data_pool 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_remove_data_pool ']' 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_remove_data_pool' 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_remove_data_pool 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:09:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:09:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_remove_data_pool 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:09:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:09:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:09:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:09:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_remove_data_pool 2024-03-23T21:09:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_remove_data_pool'] 2024-03-23T21:09:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:09:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_remove_data_pool 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:09:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:09:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:09:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:09:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:09:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:09:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_remove_data_pool 2024-03-23T21:09:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_remove_data_pool 2024-03-23T21:09:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_remove_data_pool '' 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool 2024-03-23T21:09:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_remove_data_pool 2024-03-23T21:09:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-03-23T21:09:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_remove_data_pool '..*' 2024-03-23T21:09:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-03-23T21:09:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:09:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:09:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool 2024-03-23T21:09:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:09:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:09:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:09:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:09:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test_remove_data_pool.status 2024-03-23T21:09:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position '' 2024-03-23T21:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:09:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:09:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:09:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_remove_data_pool 2024-03-23T21:09:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_remove_data_pool: 2024-03-23T21:09:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 17ef781e-bec6-4571-b645-de7d35efc02b 2024-03-23T21:09:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:09:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:09:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:09:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:09:33 2024-03-23T21:09:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:09:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:09:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:09:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-03-23T21:09:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:09:37' 2024-03-23T21:09:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:09:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:09:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:09:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:09:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:09:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:09:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:09:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:09:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:09:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:09:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:09:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:09:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:09:33' 2024-03-23T21:09:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_remove_data_pool up+replaying '2024-03-23 21:09:33' 2024-03-23T21:09:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:09:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:09:33' 2024-03-23T21:09:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:09:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:09:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:09:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK442894296mirror.0smithi20418.2.2-954-g82316b04trueOK42934299mirror.1smithi20418.2.2-954-g82316b04falseOKtesta690f084-85c3-4b19-8ea7-4e4ff8d04c66up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42894296mirror.0smithi2042024-03-23 21:09:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:09:37test2a531096b-315e-4e71-9418-5786990aade9up+replayingreplaying, {"bytes_per_second":35.0,"entries_behind_primary":0,"entries_per_second":0.4,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42934299mirror.1smithi2042024-03-23 21:09:27cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:09:37test4360e8a7c-d090-48c6-919a-1a1fd81371bbup+replayingreplaying, {"bytes_per_second":306.0,"entries_behind_primary":0,"entries_per_second":3.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42894296mirror.0smithi2042024-03-23 21:09:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:09:37test_remove_data_pool17ef781e-bec6-4571-b645-de7d35efc02bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42934299mirror.1smithi2042024-03-23 21:09:33cluster2e9ef87d4-e149-495b-ba3d-134c4a685598up+stoppedlocal image is primary2024-03-23 21:09:37' 2024-03-23T21:09:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:09:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_remove_data_pool'\'']/last_update' 2024-03-23T21:09:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:09:33' 2024-03-23T21:09:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_remove_data_pool'\'']/state' 2024-03-23T21:09:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:09:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:09:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:09:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:09:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-03-23T21:09:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:09:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it 2024-03-23T21:09:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'pool_to_remove' does not exist 2024-03-23T21:09:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-03-23T21:09:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:09:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it 2024-03-23T21:09:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'pool_to_remove' does not exist 2024-03-23T21:09:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_remove_data_pool 2024-03-23T21:09:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:09:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_remove_data_pool 2024-03-23T21:09:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_remove_data_pool 2024-03-23T21:09:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_remove_data_pool 2024-03-23T21:09:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 100% complete...done. 2024-03-23T21:09:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test_remove_data_pool deleted 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test_remove_data_pool 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test_remove_data_pool 2024-03-23T21:09:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory 2024-03-23T21:09:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:09:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:09:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:09:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test_remove_data_pool deleted '' 2024-03-23T21:09:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:09:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:09:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-03-23T21:09:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:09:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:09:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:09:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:09:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test_remove_data_pool 2024-03-23T21:09:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:09:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:09:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-03-23T21:09:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test_remove_data_pool 2024-03-23T21:09:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:09:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory 2024-03-23T21:09:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:09:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:09:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:09:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:09:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16 2024-03-23T21:09:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2024-03-23T21:09:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:09:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:09:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:09:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:09:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:09:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:09:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:09:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:09:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:09:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:09:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:09:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:09:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:09:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:09:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:09:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:09:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:09:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:09:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:09:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:09:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:09:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:09:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:09:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:09:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:09:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:09:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:09:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:09:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:09:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:09:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:09:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:09:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:09:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:09:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:09:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:09:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:09:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:09:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:09:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:09:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:09:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:09:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:09:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:09:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:09:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:09:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:09:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:09:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:09:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:09:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:09:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:09:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:09:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:09:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:09:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:09:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:09:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:09:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:09:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:09:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:09:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:09:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:10:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:10:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:10:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:10:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:10:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:10:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:10:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:10:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:10:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:10:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:10:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:10:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:10:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:10:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:10:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:10:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:10:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:10:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:10:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:10:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:10:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:10:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:10:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:10:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:10:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:10:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:10:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:10:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:10:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:16.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:10:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:10:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:10:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:10:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:10:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:10:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:10:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:10:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:10:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:10:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:10:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:10:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:10:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:10:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:10:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:10:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:10:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:10:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:10:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:10:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:10:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:10:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:10:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:10:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:10:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:10:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:10:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:10:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:10:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:10:33.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:10:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:10:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:10:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:10:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:10:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:10:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:10:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:10:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:10:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:10:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:10:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:10:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:10:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:10:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:10:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:10:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:10:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:10:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:10:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:10:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:10:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:10:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:10:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:10:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:10:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:10:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:10:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:10:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:10:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:10:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:10:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:10:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:10:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:10:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:10:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:10:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:10:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:10:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:10:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:10:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:10:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:10:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:10:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:10:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:10:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:10:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:10:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:10:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 16 2024-03-23T21:11:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:11:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:11:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:11:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:11:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:11:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:11:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:11:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:11:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:11:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:11:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:11:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:11:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:11:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:11:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:11:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 16 2024-03-23T21:11:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:11:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:11:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:11:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:11:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:11:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:11:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:11:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:11:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:11:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:11:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:11:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:11:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:11:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:11:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:11:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:11:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:11:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:11:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:11:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:11:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:11:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:11:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:11:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:11:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:11:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:11:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:11:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:11:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:11:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:11:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:11:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:11:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:11:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:11:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:11:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:11:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:11:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:11:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:11:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:11:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:11:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:11:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:11:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:11:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:11:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:11:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:11:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:11:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:11:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:11:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:11:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:11:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:11:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:11:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:11:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:11:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:11:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:11:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:11:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:11:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:11:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:11:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:11:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:11:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:11:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:11:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:11:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:11:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:11:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:11:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:11:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:11:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:11:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:11:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:11:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:11:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:11:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:11:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:11:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:11:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:11:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:11:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:11:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:11:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:11:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:11:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:11:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:11:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:11:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:11:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:11:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:11:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:11:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:11:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:11:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:11:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:11:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:11:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:11:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:12:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:12:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:12:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:12:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:12:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:12:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:12:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:03.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:12:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:12:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:12:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:12:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:12:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:12:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:12:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:12:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:12:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:12:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:12:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:12:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:12:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:12:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:12:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:12:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:12:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:12:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:12:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:12:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:12:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:12:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:12:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:12:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:12:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:12:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:12:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:12:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:12:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:12:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:12:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:12:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:12:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:12:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:12:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:12:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:12:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:12:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:12:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:12:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:12:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:12:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:12:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:12:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:12:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:12:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:12:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:12:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:12:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:12:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:12:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:12:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:12:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:12:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:12:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:12:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:12:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:12:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:12:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:12:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:12:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:12:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:12:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:12:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:12:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:12:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:12:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:12:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:12:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:12:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:12:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:12:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:12:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:12:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:12:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:12:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:12:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:12:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:12:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:12:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:12:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:12:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:12:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:12:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:12:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:12:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:12:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:12:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:12:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:12:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:12:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:12:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:12:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:12:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:12:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:12:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:12:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:12:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:12:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:12:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:12:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 16 2024-03-23T21:13:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:13:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:13:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:13:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:13:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:13:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:13:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:13:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:13:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:13:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:13:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:13:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:13:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:13:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:13:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:13:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:13:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:13:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:13:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:13:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:13:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:13:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:13:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:13:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:13:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 16 2024-03-23T21:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-03-23T21:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-03-23T21:13:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:13:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:13:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:13:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:13:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-03-23T21:13:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:13:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:13:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:13:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:13:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:13:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:13:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:13:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:13:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-03-23T21:13:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-03-23T21:13:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:13:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:13:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:13:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: snapshot rename' 2024-03-23T21:13:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:13:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:13:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:13:17 TEST: snapshot rename 2024-03-23T21:13:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:13:17 TEST: snapshot rename 2024-03-23T21:13:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap_rename 2024-03-23T21:13:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test2 snap_rename_0 2024-03-23T21:13:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_0 2024-03-23T21:13:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap_rename_0 2024-03-23T21:13:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-03-23T21:13:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 20 2024-03-23T21:13:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 1 - 1 2024-03-23T21:13:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_0 snap_rename_1 2024-03-23T21:13:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_0 2024-03-23T21:13:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_1 2024-03-23T21:13:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_0 mirror/test2@snap_rename_1 2024-03-23T21:13:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 2 - 1 2024-03-23T21:13:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_1 snap_rename_2 2024-03-23T21:13:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_1 2024-03-23T21:13:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_2 2024-03-23T21:13:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_1 mirror/test2@snap_rename_2 2024-03-23T21:13:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 3 - 1 2024-03-23T21:13:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_2 snap_rename_3 2024-03-23T21:13:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_2 2024-03-23T21:13:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_3 2024-03-23T21:13:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_2 mirror/test2@snap_rename_3 2024-03-23T21:13:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 4 - 1 2024-03-23T21:13:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_3 snap_rename_4 2024-03-23T21:13:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_3 2024-03-23T21:13:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_4 2024-03-23T21:13:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_3 mirror/test2@snap_rename_4 2024-03-23T21:13:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 5 - 1 2024-03-23T21:13:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_4 snap_rename_5 2024-03-23T21:13:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_4 2024-03-23T21:13:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_5 2024-03-23T21:13:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_4 mirror/test2@snap_rename_5 2024-03-23T21:13:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 6 - 1 2024-03-23T21:13:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_5 snap_rename_6 2024-03-23T21:13:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_5 2024-03-23T21:13:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_6 2024-03-23T21:13:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_5 mirror/test2@snap_rename_6 2024-03-23T21:13:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 7 - 1 2024-03-23T21:13:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_6 snap_rename_7 2024-03-23T21:13:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_6 2024-03-23T21:13:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_7 2024-03-23T21:13:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_6 mirror/test2@snap_rename_7 2024-03-23T21:13:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 8 - 1 2024-03-23T21:13:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_7 snap_rename_8 2024-03-23T21:13:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_7 2024-03-23T21:13:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_8 2024-03-23T21:13:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_7 mirror/test2@snap_rename_8 2024-03-23T21:13:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 9 - 1 2024-03-23T21:13:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_8 snap_rename_9 2024-03-23T21:13:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_8 2024-03-23T21:13:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_9 2024-03-23T21:13:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_8 mirror/test2@snap_rename_9 2024-03-23T21:13:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 10 - 1 2024-03-23T21:13:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_9 snap_rename_10 2024-03-23T21:13:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_9 2024-03-23T21:13:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_10 2024-03-23T21:13:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_9 mirror/test2@snap_rename_10 2024-03-23T21:13:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 11 - 1 2024-03-23T21:13:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_10 snap_rename_11 2024-03-23T21:13:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_10 2024-03-23T21:13:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_11 2024-03-23T21:13:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_10 mirror/test2@snap_rename_11 2024-03-23T21:13:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 12 - 1 2024-03-23T21:13:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_11 snap_rename_12 2024-03-23T21:13:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_11 2024-03-23T21:13:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_12 2024-03-23T21:13:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_11 mirror/test2@snap_rename_12 2024-03-23T21:13:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 13 - 1 2024-03-23T21:13:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_12 snap_rename_13 2024-03-23T21:13:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_12 2024-03-23T21:13:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_13 2024-03-23T21:13:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_12 mirror/test2@snap_rename_13 2024-03-23T21:13:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 14 - 1 2024-03-23T21:13:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_13 snap_rename_14 2024-03-23T21:13:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_13 2024-03-23T21:13:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_14 2024-03-23T21:13:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_13 mirror/test2@snap_rename_14 2024-03-23T21:13:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 15 - 1 2024-03-23T21:13:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_14 snap_rename_15 2024-03-23T21:13:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_14 2024-03-23T21:13:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_15 2024-03-23T21:13:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_14 mirror/test2@snap_rename_15 2024-03-23T21:13:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 16 - 1 2024-03-23T21:13:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_15 snap_rename_16 2024-03-23T21:13:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_15 2024-03-23T21:13:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_16 2024-03-23T21:13:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_15 mirror/test2@snap_rename_16 2024-03-23T21:13:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 17 - 1 2024-03-23T21:13:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_16 snap_rename_17 2024-03-23T21:13:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_16 2024-03-23T21:13:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_17 2024-03-23T21:13:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_16 mirror/test2@snap_rename_17 2024-03-23T21:13:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 18 - 1 2024-03-23T21:13:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_17 snap_rename_18 2024-03-23T21:13:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_17 2024-03-23T21:13:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_18 2024-03-23T21:13:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_17 mirror/test2@snap_rename_18 2024-03-23T21:13:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 19 - 1 2024-03-23T21:13:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_18 snap_rename_19 2024-03-23T21:13:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_18 2024-03-23T21:13:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_19 2024-03-23T21:13:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_18 mirror/test2@snap_rename_19 2024-03-23T21:13:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-03-23T21:13:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 20 - 1 2024-03-23T21:13:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_19 snap_rename_20 2024-03-23T21:13:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_19 2024-03-23T21:13:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_20 2024-03-23T21:13:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_19 mirror/test2@snap_rename_20 2024-03-23T21:13:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test2 snap_rename_20 2024-03-23T21:13:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:13:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_rename_20 2024-03-23T21:13:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:13:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:13:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:13:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-03-23T21:13:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-03-23T21:13:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-03-23T21:13:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:13:20.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:13:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-03-23T21:13:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-03-23T21:13:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-03-23T21:13:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:13:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:13:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-03-23T21:13:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-03-23T21:13:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-03-23T21:13:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:13:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:13:34.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-03-23T21:13:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-03-23T21:13:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-03-23T21:13:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-03-23T21:13:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:13:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test2': 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 3 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10cb14c121d8 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10cb14c121d8 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:09:08 2024 2024-03-23T21:13:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:09:08 2024 2024-03-23T21:13:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:09:08 2024 2024-03-23T21:13:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-03-23T21:13:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10cb14c121d8 2024-03-23T21:13:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:13:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:13:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: a531096b-315e-4e71-9418-5786990aade9 2024-03-23T21:13:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-03-23T21:13:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:13:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror test4 snap1 2024-03-23T21:13:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:13:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-03-23T21:13:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap1 2024-03-23T21:13:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror test4 snap2 2024-03-23T21:13:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:13:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-03-23T21:13:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap2 2024-03-23T21:13:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-03-23T21:13:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test2 2024-03-23T21:13:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:13:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:13:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test2 2024-03-23T21:13:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-03-23T21:13:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test2 2024-03-23T21:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing all snapshots: 33% complete... Removing all snapshots: 66% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-03-23T21:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test2 2024-03-23T21:13:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:13:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:13:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-03-23T21:13:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test4 2024-03-23T21:13:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:13:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:13:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:13:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test4 2024-03-23T21:13:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-03-23T21:13:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test4 2024-03-23T21:13:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-03-23T21:13:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test4 2024-03-23T21:13:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:13:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:13:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: disable mirror while daemon is stopped' 2024-03-23T21:13:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:13:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:13:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:13:47 TEST: disable mirror while daemon is stopped 2024-03-23T21:13:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:13:47 TEST: disable mirror while daemon is stopped 2024-03-23T21:13:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster1 2024-03-23T21:13:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:13:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:13:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:13:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:0 '' 2024-03-23T21:13:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:13:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:13:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:13:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:13:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:0 2024-03-23T21:13:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:0 2024-03-23T21:13:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:13:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:13:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:13:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:13:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:13:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:13:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:13:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-03-23T21:13:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-03-23T21:13:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:13:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:13:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-03-23T21:13:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:13:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:13:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=30783 2024-03-23T21:13:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 30783 ']' 2024-03-23T21:13:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 30783 2024-03-23T21:13:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:13:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:13:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=30783 '$2 == pid {print; exit 1}' 2024-03-23T21:13:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:ubuntu 30783 0.8 0.2 544812 66752 ? Ssl 20:55 0:09 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T21:13:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:13:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:13:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=30783 '$2 == pid {print; exit 1}' 2024-03-23T21:13:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:13:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=30783 '$2 == pid {print; exit 1}' 2024-03-23T21:13:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:13:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:13:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:13:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:13:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:13:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:13:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:13:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:13:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:13:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:0 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:13:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:13:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-03-23T21:13:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:13:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:13:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:1 '' 2024-03-23T21:13:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:13:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:1 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:1 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-03-23T21:13:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:13:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:13:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-03-23T21:13:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:13:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:13:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=30811 2024-03-23T21:13:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 30811 ']' 2024-03-23T21:13:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 30811 2024-03-23T21:13:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:13:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:13:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=30811 '$2 == pid {print; exit 1}' 2024-03-23T21:13:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:13:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=30811 '$2 == pid {print; exit 1}' 2024-03-23T21:13:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:13:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:13:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:13:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:13:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:13:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:13:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:13:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:13:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:13:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:13:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:13:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:13:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-03-23T21:13:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:13:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:13:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:13:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:13:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:13:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:13:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:1 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:13:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:13:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:13:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-03-23T21:13:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:13:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:13:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster2 2024-03-23T21:13:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:13:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:13:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster2:0 '' 2024-03-23T21:13:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:13:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:13:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:13:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:13:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster2:0 2024-03-23T21:13:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2:0 2024-03-23T21:13:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:13:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster2 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster2 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:13:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:13:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-03-23T21:13:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-03-23T21:13:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-03-23T21:13:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=34348 2024-03-23T21:13:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 34348 ']' 2024-03-23T21:13:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 34348 2024-03-23T21:13:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:13:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:13:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=34348 '$2 == pid {print; exit 1}' 2024-03-23T21:13:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:13:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=34348 '$2 == pid {print; exit 1}' 2024-03-23T21:13:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-03-23T21:13:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:13:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:13:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:13:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:13:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:13:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:13:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:13:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:13:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:13:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:13:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster2:0 2024-03-23T21:13:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:13:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:13:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:13:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:13:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:13:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:13:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:13:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:13:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:13:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-03-23T21:13:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-03-23T21:13:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-03-23T21:13:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster2:1 '' 2024-03-23T21:13:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:13:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:13:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:13:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:13:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster2:1 2024-03-23T21:13:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2:1 2024-03-23T21:13:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:13:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:13:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-03-23T21:13:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster2 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster2 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:13:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-03-23T21:13:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-03-23T21:13:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-03-23T21:13:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=34376 2024-03-23T21:13:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 34376 ']' 2024-03-23T21:13:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 34376 2024-03-23T21:13:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:13:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:13:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=34376 '$2 == pid {print; exit 1}' 2024-03-23T21:13:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:13:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:13:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=34376 '$2 == pid {print; exit 1}' 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:13:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:13:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:13:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:13:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:13:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:13:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster2:1 2024-03-23T21:13:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:13:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:13:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:13:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-03-23T21:13:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-03-23T21:13:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-03-23T21:13:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-03-23T21:13:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=image 2024-03-23T21:13:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-03-23T21:13:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:note: changing mirroring mode from pool to image 2024-03-23T21:13:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror test 2024-03-23T21:13:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:13:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:13:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test 2024-03-23T21:13:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:13:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:13:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:13:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:13:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:13:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:13:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c853a64e94 2024-03-23T21:13:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c853a64e94 2024-03-23T21:13:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c853a64e94 2024-03-23T21:13:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:13:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:13:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2024-03-23T21:13:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:13:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:13:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2024-03-23T21:13:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:13:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:13:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:13:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:13:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T21:13:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:13:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2024-03-23T21:13:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:13:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:13:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:13:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:13:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T21:13:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-03-23T21:13:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:13:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:13:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:13:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:13:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:13:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:13:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:13:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:13:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:13:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:13:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:13:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10c853a64e94 2024-03-23T21:13:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:13:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10c853a64e94 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c853a64e94 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:13:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:13:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:13:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:13:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:13:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c853a64e94 2024-03-23T21:13:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c853a64e94 2024-03-23T21:13:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c853a64e94 -o 10c853a64e94 = 10c853a64e94 2024-03-23T21:13:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:13:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:13:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:13:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:13:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10c853a64e94 2024-03-23T21:13:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:13:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:13:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:13:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c853a64e94 2024-03-23T21:13:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:13:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:13:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:13:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:13:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:13:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:13:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:13:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c853a64e94 2024-03-23T21:13:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c853a64e94 2024-03-23T21:13:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c853a64e94 -o 10c853a64e94 = 10c853a64e94 2024-03-23T21:13:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:13:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:13:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:13:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:13:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10c853a64e94 2024-03-23T21:13:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:13:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:13:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:13:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:13:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c853a64e94 2024-03-23T21:13:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:13:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:13:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:13:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:13:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:13:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:13:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:13:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:13:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c853a64e94 2024-03-23T21:13:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c853a64e94 2024-03-23T21:13:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c853a64e94 -o 10c853a64e94 = 10c853a64e94 2024-03-23T21:13:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:13:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:13:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:13:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:14:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10c853a64e94 2024-03-23T21:14:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c853a64e94 2024-03-23T21:14:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror pool 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=pool 2024-03-23T21:14:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool 2024-03-23T21:14:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:note: changing mirroring mode from image to pool 2024-03-23T21:14:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_journaling cluster2 mirror test 2024-03-23T21:14:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 feature enable mirror/test journaling 2024-03-23T21:14:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test present 2024-03-23T21:14:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:14:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:14:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-03-23T21:14:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:14:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-03-23T21:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:14:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:14:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-03-23T21:14:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:14:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:14:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-03-23T21:14:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16aa67bed7ec 2024-03-23T21:14:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aa67bed7ec 2024-03-23T21:14:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 16aa67bed7ec 2024-03-23T21:14:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:14:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:14:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:14:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:14:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:14:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:14:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:14:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:14:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:14:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:14:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: non-default namespace image mirroring' 2024-03-23T21:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:10 TEST: non-default namespace image mirroring 2024-03-23T21:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:10 TEST: non-default namespace image mirroring 2024-03-23T21:14:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay' 2024-03-23T21:14:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:10 - replay 2024-03-23T21:14:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:10 - replay 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror/ns1 test 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:14:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:14:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns1/test 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror/ns2 test 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:14:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns2/test 2024-03-23T21:14:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror/ns2 test journal 2024-03-23T21:14:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=journal 2024-03-23T21:14:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/ns2/test journal 2024-03-23T21:14:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2024-03-23T21:14:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/ns2/test 2024-03-23T21:14:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2024-03-23T21:14:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-03-23T21:14:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-03-23T21:14:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 0 2024-03-23T21:14:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 17a16f63e835 2024-03-23T21:14:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.17a16f63e835 2024-03-23T21:14:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-03-23T21:14:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Sat Mar 23 21:14:10 2024 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Sat Mar 23 21:14:10 2024 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Sat Mar 23 21:14:10 2024 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 17a16f63e835 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: c34bd062-a1c2-4033-acc4-fa259e8b5848 2024-03-23T21:14:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test started 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror/ns1 test started 2024-03-23T21:14:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:14:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test 2024-03-23T21:14:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2024-03-23T21:14:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2024-03-23T21:14:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2024-03-23T21:14:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2024-03-23T21:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:14:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2024-03-23T21:14:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2024-03-23T21:14:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:13.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2024-03-23T21:14:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2024-03-23T21:14:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T21:14:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2024-03-23T21:14:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2024-03-23T21:14:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:14:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:14:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:14:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:14:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:14:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:14:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:14:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror/ns2 test 2024-03-23T21:14:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror/ns2 test started 2024-03-23T21:14:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:14:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror/ns2 test started 2024-03-23T21:14:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:14:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns2/test 2024-03-23T21:14:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns2/test 2024-03-23T21:14:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns2/test 2024-03-23T21:14:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:14:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:14:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:14:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:14:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:14:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:14:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:14:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror/ns1 test 100 2024-03-23T21:14:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:14:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:14:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:14:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:14:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:14:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:14:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.37 bytes/sec: 8.1 MiB/s 2024-03-23T21:14:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror/ns2 test 100 2024-03-23T21:14:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:14:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:14:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:14:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/ns2/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:14:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:14:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:14:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2083.37 bytes/sec: 8.1 MiB/s 2024-03-23T21:14:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 test 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:14:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns1 test 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:14:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:14:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2024-03-23T21:14:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2024-03-23T21:14:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2024-03-23T21:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2024-03-23T21:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:14:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2024-03-23T21:14:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test.status 2024-03-23T21:14:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2024-03-23T21:14:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:14:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2024-03-23T21:14:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test.status 2024-03-23T21:14:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2024-03-23T21:14:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-03-23T21:14:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:14:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:14:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2024-03-23T21:14:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:14:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2024-03-23T21:14:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:14:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:14:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:14:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-03-23T21:14:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:14:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:14:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:14:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-03-23T21:14:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:14:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:14:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T21:14:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:14:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:14:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:14:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=3 2024-03-23T21:14:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 1 -a 101 -le 3 2024-03-23T21:14:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:14:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:14:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns1 test 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2024-03-23T21:14:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:14:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:14:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:14:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2024-03-23T21:14:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2024-03-23T21:14:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2024-03-23T21:14:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2024-03-23T21:14:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:14:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2024-03-23T21:14:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test.status 2024-03-23T21:14:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2024-03-23T21:14:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98], [object_number=1, tag_tid=2, entry_tid=97]]], state=connected] 2024-03-23T21:14:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test.status 2024-03-23T21:14:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2024-03-23T21:14:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98], [object_number=1, tag_tid=2, entry_tid=97]]], state=connected] 2024-03-23T21:14:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=2, entry_tid=100' 2024-03-23T21:14:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=0, tag_tid=2, entry_tid=100' 2024-03-23T21:14:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=2, entry_tid=100' '!=' 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:14:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:14:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=2, entry_tid=100' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-03-23T21:14:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=2, entry_tid=100' 2024-03-23T21:14:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:14:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:14:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-03-23T21:14:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=2, entry_tid=100' 2024-03-23T21:14:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:14:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:14:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=2 2024-03-23T21:14:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:14:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:14:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-03-23T21:14:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=2, entry_tid=100' 2024-03-23T21:14:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:14:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:14:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=100 2024-03-23T21:14:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 -a 101 -le 100 2024-03-23T21:14:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:14:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:14:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns1 test 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:14:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:14:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2024-03-23T21:14:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2024-03-23T21:14:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2024-03-23T21:14:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2024-03-23T21:14:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:14:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2024-03-23T21:14:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test.status 2024-03-23T21:14:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2024-03-23T21:14:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2024-03-23T21:14:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test.status 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2024-03-23T21:14:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.status 2024-03-23T21:14:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns2 test 2024-03-23T21:14:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:14:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns2 test 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:14:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:14:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns2 test 2024-03-23T21:14:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns2/test' 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns2/test 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns2/test 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns2/test 2024-03-23T21:14:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns2 test 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns2 test '' 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:14:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns2-test.status 2024-03-23T21:14:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns2-test.status 2024-03-23T21:14:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.status 2024-03-23T21:14:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns2/test 2024-03-23T21:14:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.status 2024-03-23T21:14:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.status 2024-03-23T21:14:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns2 test 2024-03-23T21:14:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-03-23T21:14:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns2 test '..*' 2024-03-23T21:14:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-03-23T21:14:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:14:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns2-test.status 2024-03-23T21:14:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns2-test.status 2024-03-23T21:14:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.status 2024-03-23T21:14:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns2/test 2024-03-23T21:14:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.status 2024-03-23T21:14:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.status 2024-03-23T21:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position 2024-03-23T21:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position '' 2024-03-23T21:14:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test 2024-03-23T21:14:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:14:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 3ad32aa1-34b7-4655-9335-cbe751165835 2024-03-23T21:14:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:14:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:14:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:14:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:14:14 2024-03-23T21:14:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:14:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:14:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:14:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:14:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:14:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:14:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:14:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:14:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:14:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:14:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:14:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:14:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:14:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:14:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:14' 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test up+replaying '2024-03-23 21:14:14' 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:14:14' 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2024-03-23T21:14:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK157945802mirror.0smithi20418.2.2-954-g82316b04trueOK57985806mirror.1smithi20418.2.2-954-g82316b04falseOKtest3ad32aa1-34b7-4655-9335-cbe751165835up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57945802mirror.0smithi2042024-03-23 21:14:14cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:14:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:14:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:14' 2024-03-23T21:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:14:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:14:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:14:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position '' 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns2/test 2024-03-23T21:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: c34bd062-a1c2-4033-acc4-fa259e8b5848 2024-03-23T21:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:14:14 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:14:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:14:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:14:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:14:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:14:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:14:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:14:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:14:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:14' 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns2 test up+replaying '2024-03-23 21:14:14' 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:14:14' 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns2 --verbose --format xml 2024-03-23T21:14:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK157945802mirror.0smithi20418.2.2-954-g82316b04trueOK57985806mirror.1smithi20418.2.2-954-g82316b04falseOKtestc34bd062-a1c2-4033-acc4-fa259e8b5848up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57945802mirror.0smithi2042024-03-23 21:14:14cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:14:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:14:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:14:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:14' 2024-03-23T21:14:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:14:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:14:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:14:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror/ns1 test 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror/ns1-test.export 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror/ns1-test.export 2024-03-23T21:14:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.export 2024-03-23T21:14:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror/ns1-test.export 2024-03-23T21:14:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror/ns1-test.export 2024-03-23T21:14:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test.export 2024-03-23T21:14:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test.export 2024-03-23T21:14:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/ns1/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.export 2024-03-23T21:14:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/ns1/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test.export 2024-03-23T21:14:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test.export 2024-03-23T21:14:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test.export 2024-03-23T21:14:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror/ns2 test 2024-03-23T21:14:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:14:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror/ns2-test.export 2024-03-23T21:14:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror/ns2-test.export 2024-03-23T21:14:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.export 2024-03-23T21:14:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror/ns2-test.export 2024-03-23T21:14:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror/ns2-test.export 2024-03-23T21:14:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns2-test.export 2024-03-23T21:14:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns2-test.export 2024-03-23T21:14:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/ns2/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.export 2024-03-23T21:14:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/ns2/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns2-test.export 2024-03-23T21:14:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns2-test.export 2024-03-23T21:14:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns2-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns2-test.export 2024-03-23T21:14:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - disable mirroring / delete image' 2024-03-23T21:14:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:22 - disable mirroring / delete image 2024-03-23T21:14:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:22 - disable mirroring / delete image 2024-03-23T21:14:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror/ns1 test 2024-03-23T21:14:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:14:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror/ns1 test 2024-03-23T21:14:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test 2024-03-23T21:14:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test 2024-03-23T21:14:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:14:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror/ns2 test 2024-03-23T21:14:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/ns2/test 2024-03-23T21:14:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2024-03-23T21:14:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror/ns1 test deleted 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror/ns1 test 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2024-03-23T21:14:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:14:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror/ns1 test deleted '' 2024-03-23T21:14:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror/ns1 test 2024-03-23T21:14:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2024-03-23T21:14:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror/ns2 test deleted 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror/ns2 test 2024-03-23T21:14:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-03-23T21:14:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test 2024-03-23T21:14:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:14:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror/ns2 test deleted '' 2024-03-23T21:14:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror/ns2 test 2024-03-23T21:14:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-03-23T21:14:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test 2024-03-23T21:14:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:14:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror/ns2 test 2024-03-23T21:14:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:14:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror/ns2 test 2024-03-23T21:14:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-03-23T21:14:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns2/test 2024-03-23T21:14:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/ns2/test 2024-03-23T21:14:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:14:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - data pool' 2024-03-23T21:14:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:23 - data pool 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:23 - data pool 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ dp_image=test_data_pool 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror/ns1 test_data_pool 128 --data-pool mirror_parent 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-03-23T21:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/ns1/test_data_pool 2024-03-23T21:14:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_data_pool cluster2 mirror/ns1 test_data_pool 2024-03-23T21:14:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:14:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 info mirror/ns1/test_data_pool 2024-03-23T21:14:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk '$1 == "data_pool:" {print $2}' 2024-03-23T21:14:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ data_pool=mirror_parent 2024-03-23T21:14:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test mirror_parent = mirror_parent 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test_data_pool 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test_data_pool started 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror/ns1 test_data_pool started 2024-03-23T21:14:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:14:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-03-23T21:14:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-03-23T21:14:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:14:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-03-23T21:14:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-03-23T21:14:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T21:14:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-03-23T21:14:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:14:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2024-03-23T21:14:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:14:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:14:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:14:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:14:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:14:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:14:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:14:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_data_pool cluster1 mirror/ns1 test_data_pool 2024-03-23T21:14:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:14:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk '$1 == "data_pool:" {print $2}' 2024-03-23T21:14:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 info mirror/ns1/test_data_pool 2024-03-23T21:14:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ data_pool=mirror_parent 2024-03-23T21:14:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test mirror_parent = mirror_parent 2024-03-23T21:14:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror/ns1 test_data_pool 100 2024-03-23T21:14:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-03-23T21:14:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:14:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:14:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-03-23T21:14:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-03-23T21:14:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:14:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1666.7 bytes/sec: 6.5 MiB/s 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:14:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:14:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:14:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:14:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:14:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2024-03-23T21:14:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns1/test_data_pool' 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2024-03-23T21:14:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2024-03-23T21:14:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-03-23T21:14:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:14:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:14:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:14:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:14:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2024-03-23T21:14:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2024-03-23T21:14:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:14:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '' 2024-03-23T21:14:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:14:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:14:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2024-03-23T21:14:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2024-03-23T21:14:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.status 2024-03-23T21:14:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2024-03-23T21:14:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.status 2024-03-23T21:14:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.status 2024-03-23T21:14:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2024-03-23T21:14:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '..*' 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2024-03-23T21:14:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.status 2024-03-23T21:14:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2024-03-23T21:14:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.status 2024-03-23T21:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-03-23T21:14:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.status 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying primary_position 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying primary_position '' 2024-03-23T21:14:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test_data_pool 2024-03-23T21:14:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_data_pool: 2024-03-23T21:14:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a5eeae78-8363-45e5-8e23-6fe35fae8c99 2024-03-23T21:14:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:14:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:14:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:14:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:14:26 2024-03-23T21:14:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:14:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:14:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:14:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:14:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:14:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:14:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:14:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:14:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:14:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:14:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:14:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:14:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:14:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:14:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:26' 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test_data_pool up+replaying '2024-03-23 21:14:26' 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:14:26' 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2024-03-23T21:14:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK157945802mirror.0smithi20418.2.2-954-g82316b04trueOK57985806mirror.1smithi20418.2.2-954-g82316b04falseOKtest_data_poola5eeae78-8363-45e5-8e23-6fe35fae8c99up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57985806mirror.1smithi2042024-03-23 21:14:26cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:14:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:14:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update' 2024-03-23T21:14:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:26' 2024-03-23T21:14:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state' 2024-03-23T21:14:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:14:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:14:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror/ns1 test_data_pool 2024-03-23T21:14:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:14:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror/ns1-test_data_pool.export 2024-03-23T21:14:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror/ns1-test_data_pool.export 2024-03-23T21:14:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.export 2024-03-23T21:14:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror/ns1-test_data_pool.export 2024-03-23T21:14:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror/ns1-test_data_pool.export 2024-03-23T21:14:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test_data_pool.export 2024-03-23T21:14:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test_data_pool.export 2024-03-23T21:14:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/ns1/test_data_pool /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.export 2024-03-23T21:14:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/ns1/test_data_pool /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test_data_pool.export 2024-03-23T21:14:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test_data_pool.export 2024-03-23T21:14:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror_ns1-test_data_pool.export 2024-03-23T21:14:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror/ns1 test_data_pool 2024-03-23T21:14:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:14:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror/ns1 test_data_pool 2024-03-23T21:14:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-03-23T21:14:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-03-23T21:14:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test_data_pool 2024-03-23T21:14:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test_data_pool 2024-03-23T21:14:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:14:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: simple image resync' 2024-03-23T21:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:31 TEST: simple image resync 2024-03-23T21:14:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:31 TEST: simple image resync 2024-03-23T21:14:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror test image_id 2024-03-23T21:14:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-03-23T21:14:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''16aa67bed7ec'\''' 2024-03-23T21:14:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=16aa67bed7ec 2024-03-23T21:14:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-03-23T21:14:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 16aa67bed7ec 2024-03-23T21:14:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2024-03-23T21:14:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 16aa67bed7ec 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aa67bed7ec 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aa67bed7ec 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16aa67bed7ec 2024-03-23T21:14:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aa67bed7ec 2024-03-23T21:14:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test present 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:14:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:14:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-03-23T21:14:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:14:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-03-23T21:14:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:14:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:14:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-03-23T21:14:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16aa6c5adb23 2024-03-23T21:14:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aa6c5adb23 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 16aa6c5adb23 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:14:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:14:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:14:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T21:14:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T21:14:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T21:14:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T21:14:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:14:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:14:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:14:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:14:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:14:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:14:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:14:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-03-23T21:14:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:14:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:14:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9c870584-2c31-4abd-aa68-de0ec75a11b4 2024-03-23T21:14:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:14:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:14:36 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:14:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:14:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:14:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:14:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:14:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:14:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:14:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:14:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:14:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:14:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:36' 2024-03-23T21:14:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:14:36' 2024-03-23T21:14:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:14:36' 2024-03-23T21:14:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:14:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING157945802mirror.0smithi20418.2.2-954-g82316b04trueOK57985806mirror.1smithi20418.2.2-954-g82316b04falseOKtest9c870584-2c31-4abd-aa68-de0ec75a11b4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}57945802mirror.0smithi2042024-03-23 21:14:36cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:14:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:14:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:14:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:36' 2024-03-23T21:14:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:14:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:14:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:14:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T21:14:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:14:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T21:14:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T21:14:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T21:14:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T21:14:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T21:14:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T21:14:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: image resync while replayer is stopped' 2024-03-23T21:14:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:38 TEST: image resync while replayer is stopped 2024-03-23T21:14:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:38 TEST: image resync while replayer is stopped 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/test 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/test 2024-03-23T21:14:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/test 2024-03-23T21:14:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-03-23T21:14:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-03-23T21:14:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:14:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:14:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:14:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:14:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:14:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:14:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:14:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:14:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:14:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror test image_id 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''16aa6c5adb23'\''' 2024-03-23T21:14:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=16aa6c5adb23 2024-03-23T21:14:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-03-23T21:14:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 16aa6c5adb23 2024-03-23T21:14:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2024-03-23T21:14:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-03-23T21:14:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2024-03-23T21:14:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:14:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:14:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:14:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 16aa6c5adb23 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aa6c5adb23 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aa6c5adb23 2024-03-23T21:14:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16aa6c5adb23 2024-03-23T21:14:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aa6c5adb23 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-03-23T21:14:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:14:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:14:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2024-03-23T21:14:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:14:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:14:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:14:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2024-03-23T21:14:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test present 2024-03-23T21:14:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:14:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=16aab5af2fef 2024-03-23T21:14:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present 16aab5af2fef 2024-03-23T21:14:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aab5af2fef 2024-03-23T21:14:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16aab5af2fef 2024-03-23T21:14:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aab5af2fef 2024-03-23T21:14:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16aab5af2fef -o 16aab5af2fef = 16aab5af2fef 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:14:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:14:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:14:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:14:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T21:14:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-03-23T21:14:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-03-23T21:14:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:14:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:14:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:14:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:14:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:14:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:14:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:14:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:14:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:14:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:14:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-03-23T21:14:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-03-23T21:14:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9c870584-2c31-4abd-aa68-de0ec75a11b4 2024-03-23T21:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-03-23T21:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:14:42 2024-03-23T21:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:14:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:14:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:14:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:14:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:14:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:14:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:14:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:14:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-03-23T21:14:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:14:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:14:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:14:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:14:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:14:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:14:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:42' 2024-03-23T21:14:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:14:42' 2024-03-23T21:14:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:14:42' 2024-03-23T21:14:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:14:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING157945802mirror.0smithi20418.2.2-954-g82316b04trueOK57985806mirror.1smithi20418.2.2-954-g82316b04falseOKtest9c870584-2c31-4abd-aa68-de0ec75a11b4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}57945802mirror.0smithi2042024-03-23 21:14:42cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:14:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:14:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:14:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:42' 2024-03-23T21:14:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:14:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:14:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:14:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T21:14:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:14:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T21:14:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T21:14:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T21:14:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T21:14:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T21:14:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T21:14:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: request image resync while daemon is offline' 2024-03-23T21:14:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:45 TEST: request image resync while daemon is offline 2024-03-23T21:14:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:45 TEST: request image resync while daemon is offline 2024-03-23T21:14:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster1 2024-03-23T21:14:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:14:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:0 '' 2024-03-23T21:14:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:14:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:0 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:0 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-03-23T21:14:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-03-23T21:14:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-03-23T21:14:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:14:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:14:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=64701 2024-03-23T21:14:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 64701 ']' 2024-03-23T21:14:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 64701 2024-03-23T21:14:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:14:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:14:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=64701 '$2 == pid {print; exit 1}' 2024-03-23T21:14:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:14:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:14:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=64701 '$2 == pid {print; exit 1}' 2024-03-23T21:14:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:0 2024-03-23T21:14:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-03-23T21:14:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:14:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:14:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:1 '' 2024-03-23T21:14:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:14:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:14:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:1 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:1 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-03-23T21:14:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-03-23T21:14:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-03-23T21:14:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:14:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:14:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=64729 2024-03-23T21:14:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 64729 ']' 2024-03-23T21:14:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 64729 2024-03-23T21:14:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:14:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:14:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=64729 '$2 == pid {print; exit 1}' 2024-03-23T21:14:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:14:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:14:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=64729 '$2 == pid {print; exit 1}' 2024-03-23T21:14:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:14:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:47.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:14:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:1 2024-03-23T21:14:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-03-23T21:14:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:14:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror test image_id 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''16aab5af2fef'\''' 2024-03-23T21:14:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=16aab5af2fef 2024-03-23T21:14:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-03-23T21:14:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 16aab5af2fef 2024-03-23T21:14:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2024-03-23T21:14:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-03-23T21:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2024-03-23T21:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:14:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:14:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T21:14:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2024-03-23T21:14:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:14:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:14:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:14:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:14:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:14:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:14:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:14:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T21:14:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 16aab5af2fef 2024-03-23T21:14:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:14:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aab5af2fef 2024-03-23T21:14:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aab5af2fef 2024-03-23T21:14:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16aab5af2fef 2024-03-23T21:14:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aab5af2fef 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16aab5af2fef 2024-03-23T21:14:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aab5af2fef 2024-03-23T21:14:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16aab5af2fef -o 16aab5af2fef = 16aab5af2fef 2024-03-23T21:14:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:14:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:14:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16aab5af2fef 2024-03-23T21:14:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aab5af2fef 2024-03-23T21:14:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16aab5af2fef 2024-03-23T21:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aab5af2fef 2024-03-23T21:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16aab5af2fef -o 16aab5af2fef = 16aab5af2fef 2024-03-23T21:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:14:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16aab5af2fef 2024-03-23T21:14:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aab5af2fef 2024-03-23T21:14:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16aab5af2fef 2024-03-23T21:14:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16aab5af2fef 2024-03-23T21:14:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16aab5af2fef -o 16aab5af2fef = 16aab5af2fef 2024-03-23T21:14:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:14:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:14:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:14:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16aab5af2fef 2024-03-23T21:14:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:14:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16aab5af2fef 2024-03-23T21:14:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=173f2025b20d 2024-03-23T21:14:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 173f2025b20d 2024-03-23T21:14:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16aab5af2fef -o 16aab5af2fef = 173f2025b20d 2024-03-23T21:14:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:14:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test present 2024-03-23T21:14:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=173f2025b20d 2024-03-23T21:14:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:14:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:14:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present 173f2025b20d 2024-03-23T21:14:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:14:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=173f2025b20d 2024-03-23T21:14:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:14:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:14:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-03-23T21:14:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:14:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-03-23T21:14:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-03-23T21:14:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:14:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=173f2025b20d 2024-03-23T21:14:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 173f2025b20d 2024-03-23T21:14:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 173f2025b20d -o 173f2025b20d = 173f2025b20d 2024-03-23T21:14:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:14:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:14:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-03-23T21:14:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:14:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-03-23T21:14:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-03-23T21:14:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:14:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:14:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:14:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:14:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:14:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:14:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:14:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-03-23T21:14:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-03-23T21:14:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:14:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:14:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-03-23T21:14:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9c870584-2c31-4abd-aa68-de0ec75a11b4 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:14:54 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:14:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:14:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:14:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:14:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2024-03-23T21:14:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:14:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:14:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:14:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:14:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:14:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:54' 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-03-23 21:14:54' 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:14:54' 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:14:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:14:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING159445951mirror.0smithi20418.2.2-954-g82316b04trueOK59485954mirror.1smithi20418.2.2-954-g82316b04falseOKtest9c870584-2c31-4abd-aa68-de0ec75a11b4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}59445951mirror.0smithi2042024-03-23 21:14:54cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:14:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:14:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-03-23T21:14:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:14:54' 2024-03-23T21:14:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-03-23T21:14:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:14:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:14:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:14:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:14:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-03-23T21:14:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:14:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-03-23T21:14:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-03-23T21:14:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T21:14:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-03-23T21:14:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-03-23T21:14:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export 2024-03-23T21:14:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:14:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-test.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-test.export 2024-03-23T21:14:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test 2024-03-23T21:14:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:14:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:14:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test 2024-03-23T21:14:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-03-23T21:14:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-03-23T21:14:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-03-23T21:14:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:14:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:14:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: client disconnect' 2024-03-23T21:14:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:58 TEST: client disconnect 2024-03-23T21:14:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:58 TEST: client disconnect 2024-03-23T21:14:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=laggy 2024-03-23T21:14:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror laggy 128 --journal-object-size 64K 2024-03-23T21:14:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:14:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:14:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-03-23T21:14:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-03-23T21:14:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:14:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --journal-object-size 64K mirror/laggy 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror laggy 10 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=10 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:14:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand 2024-03-23T21:14:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 40960 pattern random 2024-03-23T21:14:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:14:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 10 ops/sec: 833.347 bytes/sec: 3.3 MiB/s 2024-03-23T21:14:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay stopped after disconnect' 2024-03-23T21:14:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:14:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:14:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:14:58 - replay stopped after disconnect 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:14:58 - replay stopped after disconnect 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:14:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy started 2024-03-23T21:14:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:14:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:14:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:14:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:14:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:14:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:14:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-03-23T21:14:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:14:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:14:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:14:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:14:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:14:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:14:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:14:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:14:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:14:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:14:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:14:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:14:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:14:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:14:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:14:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:14:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:14:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:14:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:14:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:14:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:14:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:14:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:14:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:14:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:14:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:14:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:14:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:14:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:14:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:14:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:14:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:14:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:14:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:15:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-03-23T21:15:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:15:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:15:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:15:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:15:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:15:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:15:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:15:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:15:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:15:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:15:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:15:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:15:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:15:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:15:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' 2024-03-23T21:15:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disconnect_image cluster2 mirror laggy 2024-03-23T21:15:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2024-03-23T21:15:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:client b3e423e8-28b3-4966-87da-be3892bfa4ca disconnected 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:15:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-03-23T21:15:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=disconnected] 2024-03-23T21:15:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:15:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2024-03-23T21:15:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:15:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:15:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:15:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:15:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:15:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:15:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2024-03-23T21:15:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-03-23T21:15:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2024-03-23T21:15:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-03-23T21:15:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2024-03-23T21:15:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='laggy: 2024-03-23T21:15:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 331e8f56-2e8c-4fb4-9cc9-097bb366e937 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: disconnected 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:03 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-03-23T21:15:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+error 2024-03-23T21:15:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*disconnected' 2024-03-23T21:15:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: disconnected 2024-03-23T21:15:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:15:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-03-23T21:15:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:15:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-03-23T21:15:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:15:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:15:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:15:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:15:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:03' 2024-03-23T21:15:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-03-23 21:15:03' 2024-03-23T21:15:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:06.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:15:03' 2024-03-23T21:15:06.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:06.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:06.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:15:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING159445951mirror.0smithi20418.2.2-954-g82316b04trueOK59485954mirror.1smithi20418.2.2-954-g82316b04falseOKlaggy331e8f56-2e8c-4fb4-9cc9-097bb366e937up+errordisconnected59485954mirror.1smithi2042024-03-23 21:15:03cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:15:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:15:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2024-03-23T21:15:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:03' 2024-03-23T21:15:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2024-03-23T21:15:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+error 2024-03-23T21:15:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-03-23T21:15:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+error 2024-03-23T21:15:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-03-23T21:15:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay started after resync requested' 2024-03-23T21:15:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:15:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:15:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:15:06 - replay started after resync requested 2024-03-23T21:15:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:15:06 - replay started after resync requested 2024-03-23T21:15:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror laggy image_id 2024-03-23T21:15:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-03-23T21:15:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''17421c0b53bc'\''' 2024-03-23T21:15:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=17421c0b53bc 2024-03-23T21:15:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-03-23T21:15:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 17421c0b53bc 2024-03-23T21:15:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/laggy 2024-03-23T21:15:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-03-23T21:15:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 17421c0b53bc 2024-03-23T21:15:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:15:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=17421c0b53bc 2024-03-23T21:15:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 17421c0b53bc 2024-03-23T21:15:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:15:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 17421c0b53bc 2024-03-23T21:15:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:15:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=17421c0b53bc 2024-03-23T21:15:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=17421c0b53bc 2024-03-23T21:15:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 17421c0b53bc 2024-03-23T21:15:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 17421c0b53bc -o 17421c0b53bc = 17421c0b53bc 2024-03-23T21:15:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:15:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:15:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 17421c0b53bc 2024-03-23T21:15:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:15:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=17421c0b53bc 2024-03-23T21:15:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=17421c0b53bc 2024-03-23T21:15:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 17421c0b53bc 2024-03-23T21:15:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 17421c0b53bc -o 17421c0b53bc = 17421c0b53bc 2024-03-23T21:15:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:15:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:15:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:15:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 17421c0b53bc 2024-03-23T21:15:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:15:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=17421c0b53bc 2024-03-23T21:15:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:15:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:15:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy present 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:15:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:15:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-03-23T21:15:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:15:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:15:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:15:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:15:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-03-23T21:15:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:15:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=1742cf548955 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1742cf548955 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 1742cf548955 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:15:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy started 2024-03-23T21:15:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:15:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:15:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:15:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-03-23T21:15:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:15:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:15:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:15:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:15:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:15:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-03-23T21:15:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:15:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:15:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:15:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:15:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:15:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:15:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T21:15:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T21:15:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T21:15:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:15:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T21:15:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T21:15:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T21:15:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' -a 'object_number=3, tag_tid=3, entry_tid=3' = 'object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:15:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T21:15:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T21:15:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T21:15:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror laggy 2024-03-23T21:15:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:15:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-laggy.export 2024-03-23T21:15:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-laggy.export 2024-03-23T21:15:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export 2024-03-23T21:15:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-laggy.export 2024-03-23T21:15:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-laggy.export 2024-03-23T21:15:16.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export 2024-03-23T21:15:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:15:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:15:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - disconnected after max_concurrent_object_sets reached' 2024-03-23T21:15:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:15:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:15:16 - disconnected after max_concurrent_object_sets reached 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:15:16 - disconnected after max_concurrent_object_sets reached 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/laggy 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/laggy 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/laggy 2024-03-23T21:15:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror stop : stop rbd mirror 2024-03-23T21:15:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror/laggy 2024-03-23T21:15:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror/laggy 2024-03-23T21:15:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2024-03-23T21:15:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:15:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:15:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:15:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-03-23T21:15:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:15:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:15:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:15:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:15:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:15:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:15:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:15:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-03-23T21:15:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T21:15:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-03-23T21:15:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' 2024-03-23T21:15:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_journal_max_concurrent_object_sets 1 2024-03-23T21:15:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=conf_rbd_journal_max_concurrent_object_sets 2024-03-23T21:15:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=1 2024-03-23T21:15:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_journal_max_concurrent_object_sets 1 2024-03-23T21:15:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror laggy 20 16384 2024-03-23T21:15:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=20 2024-03-23T21:15:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=16384 2024-03-23T21:15:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16384 2024-03-23T21:15:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 16384 --io-threads 1 --io-total 327680 --io-pattern rand 2024-03-23T21:15:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 16384 io_threads 1 bytes 327680 pattern random 2024-03-23T21:15:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:15:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 20 ops/sec: 625.01 bytes/sec: 9.8 MiB/s 2024-03-23T21:15:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror laggy 20 16384 2024-03-23T21:15:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=20 2024-03-23T21:15:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=16384 2024-03-23T21:15:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16384 2024-03-23T21:15:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 16384 --io-threads 1 --io-total 327680 --io-pattern rand 2024-03-23T21:15:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 16384 io_threads 1 bytes 327680 pattern random 2024-03-23T21:15:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:15:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 20 ops/sec: 416.673 bytes/sec: 6.5 MiB/s 2024-03-23T21:15:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-03-23T21:15:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=6, entry_tid=41], [object_number=16, tag_tid=6, entry_tid=40], [object_number=27, tag_tid=6, entry_tid=39], [object_number=14, tag_tid=6, entry_tid=38]]], state=connected] 2024-03-23T21:15:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=disconnected] 2024-03-23T21:15:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' 2024-03-23T21:15:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_journal_max_concurrent_object_sets 0 2024-03-23T21:15:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=conf_rbd_journal_max_concurrent_object_sets 2024-03-23T21:15:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=0 2024-03-23T21:15:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_journal_max_concurrent_object_sets 0 2024-03-23T21:15:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay is still stopped (disconnected) after restart' 2024-03-23T21:15:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:15:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:15:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:15:18 - replay is still stopped '(disconnected)' after restart 2024-03-23T21:15:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:15:18 - replay is still stopped (disconnected) after restart 2024-03-23T21:15:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror start mirror/laggy 2024-03-23T21:15:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/laggy 2024-03-23T21:15:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/laggy 2024-03-23T21:15:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror start : start rbd mirror 2024-03-23T21:15:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror/laggy 2024-03-23T21:15:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror/laggy 2024-03-23T21:15:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2024-03-23T21:15:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:15:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:15:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:15:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-03-23T21:15:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:15:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:15:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:15:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:15:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:15:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:15:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:15:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2024-03-23T21:15:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='laggy: 2024-03-23T21:15:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 331e8f56-2e8c-4fb4-9cc9-097bb366e937 2024-03-23T21:15:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-03-23T21:15:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: disconnected 2024-03-23T21:15:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:15:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:18 2024-03-23T21:15:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-03-23T21:15:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+error 2024-03-23T21:15:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*disconnected' 2024-03-23T21:15:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: disconnected 2024-03-23T21:15:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:15:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-03-23T21:15:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:15:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-03-23T21:15:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:15:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:15:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:15:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:15:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:18' 2024-03-23T21:15:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-03-23 21:15:18' 2024-03-23T21:15:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:15:18' 2024-03-23T21:15:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:15:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING159445951mirror.0smithi20418.2.2-954-g82316b04trueOK59485954mirror.1smithi20418.2.2-954-g82316b04falseOKlaggy331e8f56-2e8c-4fb4-9cc9-097bb366e937up+errordisconnected59485954mirror.1smithi2042024-03-23 21:15:18cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:15:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:15:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2024-03-23T21:15:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:18' 2024-03-23T21:15:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2024-03-23T21:15:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+error 2024-03-23T21:15:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-03-23T21:15:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+error 2024-03-23T21:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-03-23T21:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay started after resync requested' 2024-03-23T21:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:15:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:15:21 - replay started after resync requested 2024-03-23T21:15:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:15:21 - replay started after resync requested 2024-03-23T21:15:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror laggy image_id 2024-03-23T21:15:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-03-23T21:15:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''1742cf548955'\''' 2024-03-23T21:15:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=1742cf548955 2024-03-23T21:15:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-03-23T21:15:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 1742cf548955 2024-03-23T21:15:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/laggy 2024-03-23T21:15:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 1742cf548955 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=1742cf548955 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1742cf548955 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:15:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 1742cf548955 2024-03-23T21:15:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:15:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=1742cf548955 2024-03-23T21:15:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=1742cf548955 2024-03-23T21:15:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1742cf548955 2024-03-23T21:15:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 1742cf548955 -o 1742cf548955 = 1742cf548955 2024-03-23T21:15:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:15:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:15:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 1742cf548955 2024-03-23T21:15:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:15:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=1742cf548955 2024-03-23T21:15:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=1742cf548955 2024-03-23T21:15:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1742cf548955 2024-03-23T21:15:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 1742cf548955 -o 1742cf548955 = 1742cf548955 2024-03-23T21:15:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:15:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-03-23T21:15:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 1742cf548955 2024-03-23T21:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:15:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=1742cf548955 2024-03-23T21:15:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:15:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:15:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy present 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:15:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:15:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-03-23T21:15:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:15:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:15:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:15:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-03-23T21:15:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:15:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:15:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:15:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:15:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-03-23T21:15:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:15:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=17423ee53986 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 17423ee53986 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 17423ee53986 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2024-03-23T21:15:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:15:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy started 2024-03-23T21:15:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:15:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:15:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:15:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:15:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:15:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:15:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:15:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:15:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 3 2024-03-23T21:15:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-03-23T21:15:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0], [object_number=27, tag_tid=6, entry_tid=39], [object_number=14, tag_tid=6, entry_tid=38]]], state=connected] 2024-03-23T21:15:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=8, entry_tid=3' 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 3 2024-03-23T21:15:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-03-23T21:15:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0], [object_number=27, tag_tid=6, entry_tid=39], [object_number=14, tag_tid=6, entry_tid=38]]], state=connected] 2024-03-23T21:15:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=8, entry_tid=1' 2024-03-23T21:15:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' -a 'object_number=27, tag_tid=8, entry_tid=3' = 'object_number=25, tag_tid=8, entry_tid=1' 2024-03-23T21:15:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=8, entry_tid=1' '!=' '' 2024-03-23T21:15:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:15:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=8, entry_tid=1' = '' 2024-03-23T21:15:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=25, tag_tid=8, entry_tid=1' 2024-03-23T21:15:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3' 2024-03-23T21:15:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:15:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=8 2024-03-23T21:15:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=8, entry_tid=1' 2024-03-23T21:15:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:15:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=8 2024-03-23T21:15:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3' 2024-03-23T21:15:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:15:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-03-23T21:15:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=8, entry_tid=1' 2024-03-23T21:15:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:15:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-03-23T21:15:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 8 = 8 -a 3 -le 1 2024-03-23T21:15:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-03-23T21:15:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-03-23T21:15:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=8, entry_tid=3' 2024-03-23T21:15:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-03-23T21:15:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-03-23T21:15:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=27, tag_tid=8, entry_tid=3' 2024-03-23T21:15:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' -a 'object_number=27, tag_tid=8, entry_tid=3' = 'object_number=27, tag_tid=8, entry_tid=3' 2024-03-23T21:15:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-03-23T21:15:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-03-23T21:15:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' 2024-03-23T21:15:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror laggy 2024-03-23T21:15:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:15:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-laggy.export 2024-03-23T21:15:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-laggy.export 2024-03-23T21:15:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export 2024-03-23T21:15:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-laggy.export 2024-03-23T21:15:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-laggy.export 2024-03-23T21:15:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export 2024-03-23T21:15:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:15:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:15:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - rbd_mirroring_resync_after_disconnect config option' 2024-03-23T21:15:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:15:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:15:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:15:31 - rbd_mirroring_resync_after_disconnect config option 2024-03-23T21:15:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:15:31 - rbd_mirroring_resync_after_disconnect config option 2024-03-23T21:15:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_mirroring_resync_after_disconnect true 2024-03-23T21:15:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=conf_rbd_mirroring_resync_after_disconnect 2024-03-23T21:15:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=true 2024-03-23T21:15:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_mirroring_resync_after_disconnect true 2024-03-23T21:15:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:15:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:15:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-03-23T21:15:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-03-23T21:15:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=25, tag_tid=9, entry_tid=1' 2024-03-23T21:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-03-23T21:15:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-03-23T21:15:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=9, entry_tid=1' 2024-03-23T21:15:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=25, tag_tid=9, entry_tid=1' -a 'object_number=25, tag_tid=9, entry_tid=1' = 'object_number=25, tag_tid=9, entry_tid=1' 2024-03-23T21:15:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=17423ee53986 2024-03-23T21:15:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disconnect_image cluster2 mirror laggy 2024-03-23T21:15:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2024-03-23T21:15:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:client b3e423e8-28b3-4966-87da-be3892bfa4ca disconnected 2024-03-23T21:15:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 17423ee53986 2024-03-23T21:15:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-03-23T21:15:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=17423ee53986 2024-03-23T21:15:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 17423ee53986 2024-03-23T21:15:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:15:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 17423ee53986 2024-03-23T21:15:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-03-23T21:15:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=17423ee53986 2024-03-23T21:15:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:15:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy present 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-03-23T21:15:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-03-23T21:15:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-03-23T21:15:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-03-23T21:15:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-03-23T21:15:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-03-23T21:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-03-23T21:15:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-03-23T21:15:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-03-23T21:15:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-03-23T21:15:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-03-23T21:15:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-03-23T21:15:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-03-23T21:15:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=1742b00f77a2 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1742b00f77a2 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 1742b00f77a2 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2024-03-23T21:15:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-03-23T21:15:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy started 2024-03-23T21:15:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-03-23T21:15:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:15:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:15:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:15:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-03-23T21:15:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:15:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:15:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:15:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:15:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-03-23T21:15:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-03-23T21:15:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-03-23T21:15:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-03-23T21:15:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:15:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:15:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:34.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-03-23T21:15:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-03-23T21:15:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=10, entry_tid=1' 2024-03-23T21:15:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' -a 'object_number=27, tag_tid=10, entry_tid=3' = 'object_number=25, tag_tid=10, entry_tid=1' 2024-03-23T21:15:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=10, entry_tid=1' '!=' '' 2024-03-23T21:15:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:15:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=10, entry_tid=1' = '' 2024-03-23T21:15:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=25, tag_tid=10, entry_tid=1' 2024-03-23T21:15:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:15:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=10 2024-03-23T21:15:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=10, entry_tid=1' 2024-03-23T21:15:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:15:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=10 2024-03-23T21:15:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:15:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-03-23T21:15:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=10, entry_tid=1' 2024-03-23T21:15:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:15:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-03-23T21:15:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 10 = 10 -a 3 -le 1 2024-03-23T21:15:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' -a 'object_number=27, tag_tid=10, entry_tid=3' = 'object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror laggy 2024-03-23T21:15:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-03-23T21:15:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-laggy.export 2024-03-23T21:15:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-laggy.export 2024-03-23T21:15:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export 2024-03-23T21:15:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-laggy.export 2024-03-23T21:15:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-laggy.export 2024-03-23T21:15:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export 2024-03-23T21:15:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:15:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-03-23T21:15:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.export /tmp/tmp.TE9AI0RDtm/cluster1-mirror-laggy.export 2024-03-23T21:15:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_mirroring_resync_after_disconnect false 2024-03-23T21:15:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=conf_rbd_mirroring_resync_after_disconnect 2024-03-23T21:15:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=false 2024-03-23T21:15:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_mirroring_resync_after_disconnect false 2024-03-23T21:15:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-03-23T21:15:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-03-23T21:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=25, tag_tid=11, entry_tid=1' 2024-03-23T21:15:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-03-23T21:15:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-03-23T21:15:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=25, tag_tid=11, entry_tid=1' -a 'object_number=25, tag_tid=11, entry_tid=1' = 'object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=27, tag_tid=10, entry_tid=3' '!=' '' 2024-03-23T21:15:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:15:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=27, tag_tid=10, entry_tid=3' = '' 2024-03-23T21:15:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=11, entry_tid=1' 2024-03-23T21:15:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:15:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=11 2024-03-23T21:15:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-03-23T21:15:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=10 2024-03-23T21:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=11, entry_tid=1' 2024-03-23T21:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:15:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=1 2024-03-23T21:15:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3' 2024-03-23T21:15:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-03-23T21:15:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-03-23T21:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=3 2024-03-23T21:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 11 = 10 -a 1 -le 3 2024-03-23T21:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-03-23T21:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-03-23T21:15:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-03-23T21:15:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-03-23T21:15:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-03-23T21:15:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-03-23T21:15:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-03-23T21:15:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-03-23T21:15:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-03-23T21:15:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-03-23T21:15:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-03-23T21:15:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:15:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-03-23T21:15:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-03-23T21:15:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:15:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:15:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-03-23T21:15:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-03-23T21:15:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-03-23T21:15:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-03-23T21:15:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-03-23T21:15:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-03-23T21:15:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=25, tag_tid=11, entry_tid=1' 2024-03-23T21:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-03-23T21:15:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-03-23T21:15:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=11, entry_tid=1' 2024-03-23T21:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=25, tag_tid=11, entry_tid=1' -a 'object_number=25, tag_tid=11, entry_tid=1' = 'object_number=25, tag_tid=11, entry_tid=1' 2024-03-23T21:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disconnect_image cluster2 mirror laggy 2024-03-23T21:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2024-03-23T21:15:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:client b3e423e8-28b3-4966-87da-be3892bfa4ca disconnected 2024-03-23T21:15:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-03-23T21:15:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-03-23T21:15:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:15:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-03-23T21:15:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-03-23T21:15:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-03-23T21:15:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-03-23T21:15:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-03-23T21:15:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-03-23T21:15:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-03-23T21:15:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-03-23T21:15:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-03-23T21:15:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-03-23T21:15:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=b3e423e8-28b3-4966-87da-be3892bfa4ca, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=disconnected] 2024-03-23T21:15:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.TE9AI0RDtm/cluster2-mirror-laggy.status 2024-03-23T21:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' 2024-03-23T21:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2024-03-23T21:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2024-03-23T21:15:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:15:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-03-23T21:15:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:15:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:15:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:15:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:15:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:15:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-03-23T21:15:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-03-23T21:15:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-03-23T21:15:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-03-23T21:15:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:15:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:15:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:15:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:15:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:15:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-03-23T21:15:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:15:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:15:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2024-03-23T21:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-03-23T21:15:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2024-03-23T21:15:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='laggy: 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 331e8f56-2e8c-4fb4-9cc9-097bb366e937 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: disconnected 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:38 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-03-23T21:15:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+error 2024-03-23T21:15:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*disconnected' 2024-03-23T21:15:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: disconnected 2024-03-23T21:15:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-03-23T21:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-03-23T21:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-03-23T21:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:38' 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-03-23 21:15:38' 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:15:38' 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:15:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING159445951mirror.0smithi20418.2.2-954-g82316b04trueOK59485954mirror.1smithi20418.2.2-954-g82316b04falseOKlaggy331e8f56-2e8c-4fb4-9cc9-097bb366e937up+errordisconnected59485954mirror.1smithi2042024-03-23 21:15:38cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:15:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:15:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2024-03-23T21:15:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:38' 2024-03-23T21:15:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2024-03-23T21:15:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+error 2024-03-23T21:15:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-03-23T21:15:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+error 2024-03-23T21:15:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-03-23T21:15:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror laggy 2024-03-23T21:15:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:15:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:15:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror laggy 2024-03-23T21:15:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-03-23T21:15:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/laggy 2024-03-23T21:15:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/laggy 2024-03-23T21:15:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:15:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: split-brain' 2024-03-23T21:15:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:15:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:15:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:15:40 TEST: split-brain 2024-03-23T21:15:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:15:40 TEST: split-brain 2024-03-23T21:15:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=split-brain 2024-03-23T21:15:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror split-brain 2024-03-23T21:15:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:15:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-03-23T21:15:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-03-23T21:15:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:15:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/split-brain 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-03-23T21:15:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:15:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:15:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:15:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image split-brain: (2) No such file or directory 2024-03-23T21:15:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status= 2024-03-23T21:15:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:15:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:15:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:15:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-03-23T21:15:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:15:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:15:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d61657d5-3934-45cf-817b-cfff72cdb6f3 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:15:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:15:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:15:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-03-23T21:15:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d61657d5-3934-45cf-817b-cfff72cdb6f3 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:47 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:15:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:15:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:15:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-03-23T21:15:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:15:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:15:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:15:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:15:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:15:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:15:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:15:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:15:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:47' 2024-03-23T21:15:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2024-03-23 21:15:47' 2024-03-23T21:15:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:15:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:15:47' 2024-03-23T21:15:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:15:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING159445951mirror.0smithi20418.2.2-954-g82316b04trueOK59485954mirror.1smithi20418.2.2-954-g82316b04falseOKsplit-braind61657d5-3934-45cf-817b-cfff72cdb6f3up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}59445951mirror.0smithi2042024-03-23 21:15:47cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:15:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:15:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2024-03-23T21:15:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:47' 2024-03-23T21:15:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2024-03-23T21:15:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:15:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:15:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:15:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:15:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror split-brain --force 2024-03-23T21:15:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force=--force 2024-03-23T21:15:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/split-brain --force 2024-03-23T21:15:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-03-23T21:15:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror split-brain 2024-03-23T21:15:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror split-brain stopped 2024-03-23T21:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-03-23T21:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-03-23T21:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror split-brain stopped 2024-03-23T21:15:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-03-23T21:15:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-03-23T21:15:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-03-23T21:15:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/split-brain 2024-03-23T21:15:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-03-23T21:15:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-03-23T21:15:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-03-23T21:15:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-03-23T21:15:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-03-23T21:15:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-03-23T21:15:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-03-23T21:15:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:15:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/split-brain 2024-03-23T21:15:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:15:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:15:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-03-23T21:15:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-03-23T21:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:15:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:15:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:15:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/split-brain 2024-03-23T21:15:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-03-23T21:15:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-03-23T21:15:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-03-23T21:15:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-03-23T21:15:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-03-23T21:15:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+stopped 2024-03-23T21:15:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:15:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:15:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+stopped '' '' 2024-03-23T21:15:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:15:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-03-23T21:15:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:15:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-03-23T21:15:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d61657d5-3934-45cf-817b-cfff72cdb6f3 2024-03-23T21:15:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:15:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:15:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:15:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:48 2024-03-23T21:15:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-03-23T21:15:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-03-23T21:15:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-03-23T21:15:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-03-23T21:15:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:15:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:15:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:15:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:15:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:15:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:15:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:15:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:48' 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+stopped '2024-03-23 21:15:48' 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:15:48' 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:15:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING159445951mirror.0smithi20418.2.2-954-g82316b04trueOK59485954mirror.1smithi20418.2.2-954-g82316b04falseOKsplit-braind61657d5-3934-45cf-817b-cfff72cdb6f3up+stoppedstopped59445951mirror.0smithi2042024-03-23 21:15:48cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:15:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:15:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2024-03-23T21:15:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:48' 2024-03-23T21:15:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2024-03-23T21:15:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-03-23T21:15:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-03-23T21:15:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-03-23T21:15:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-03-23T21:15:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster1 mirror split-brain 10 2024-03-23T21:15:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=10 2024-03-23T21:15:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-03-23T21:15:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:15:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-03-23T21:15:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 bench mirror/split-brain --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand 2024-03-23T21:15:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 40960 pattern random 2024-03-23T21:15:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-03-23T21:15:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 10 ops/sec: 416.673 bytes/sec: 1.6 MiB/s 2024-03-23T21:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror split-brain 2024-03-23T21:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/split-brain 2024-03-23T21:15:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+error split-brain 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=split-brain 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2024-03-23T21:15:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=split-brain 2024-03-23T21:15:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:15:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-03-23T21:15:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d61657d5-3934-45cf-817b-cfff72cdb6f3 2024-03-23T21:15:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-03-23T21:15:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-03-23T21:15:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:15:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:48 2024-03-23T21:15:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-03-23T21:15:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:15:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:15:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2024-03-23T21:15:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=split-brain 2024-03-23T21:15:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:15:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-03-23T21:15:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d61657d5-3934-45cf-817b-cfff72cdb6f3 2024-03-23T21:15:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-03-23T21:15:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: split-brain detected 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:52 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-03-23T21:15:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+error 2024-03-23T21:15:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*split-brain' 2024-03-23T21:15:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: split-brain detected 2024-03-23T21:15:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:15:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:15:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-03-23T21:15:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-03-23T21:15:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:15:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:15:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:15:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:15:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:52' 2024-03-23T21:15:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+error '2024-03-23 21:15:52' 2024-03-23T21:15:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-03-23T21:15:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:15:52' 2024-03-23T21:15:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:15:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING159445951mirror.0smithi20418.2.2-954-g82316b04trueOK59485954mirror.1smithi20418.2.2-954-g82316b04falseOKsplit-braind61657d5-3934-45cf-817b-cfff72cdb6f3up+errorsplit-brain detected59445951mirror.0smithi2042024-03-23 21:15:52cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:15:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:15:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2024-03-23T21:15:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:15:52' 2024-03-23T21:15:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2024-03-23T21:15:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+error 2024-03-23T21:15:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-03-23T21:15:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+error 2024-03-23T21:15:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-03-23T21:15:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror split-brain image_id 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror split-brain 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=split-brain 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/split-brain 2024-03-23T21:15:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-03-23T21:15:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''173fbb2f8cde'\''' 2024-03-23T21:15:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=173fbb2f8cde 2024-03-23T21:15:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-03-23T21:15:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 173fbb2f8cde 2024-03-23T21:15:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/split-brain 2024-03-23T21:15:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-03-23T21:15:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position 2024-03-23T21:15:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:15:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:15:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:15:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-03-23T21:15:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:15:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:15:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d61657d5-3934-45cf-817b-cfff72cdb6f3 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: split-brain detected 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:52 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:15:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:15:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:15:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-03-23T21:15:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:15:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:15:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:15:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:15:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:15:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:15:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:15:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:15:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:15:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d61657d5-3934-45cf-817b-cfff72cdb6f3 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: split-brain detected 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:15:52 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:15:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:15:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:15:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:15:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-03-23T21:16:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:16:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:16:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image split-brain: (2) No such file or directory 2024-03-23T21:16:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status= 2024-03-23T21:16:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:16:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-03-23T21:16:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-03-23T21:16:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-03-23T21:16:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-03-23T21:16:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:16:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:16:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:16:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-03-23T21:16:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-03-23T21:16:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:16:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:16:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-03-23T21:16:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-03-23T21:16:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d61657d5-3934-45cf-817b-cfff72cdb6f3 2024-03-23T21:16:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-03-23T21:16:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:16:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-03-23T21:16:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-03-23 21:16:04 2024-03-23T21:16:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-03-23T21:16:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-03-23T21:16:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-03-23T21:16:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-03-23T21:16:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-03-23T21:16:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-03-23T21:16:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-03-23T21:16:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-03-23T21:16:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-03-23T21:16:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-03-23T21:16:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:16:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-03-23T21:16:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:16:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-03-23T21:16:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-03-23T21:16:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-03-23T21:16:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:16:04' 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2024-03-23 21:16:04' 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-03-23 21:16:04' 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-03-23T21:16:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-03-23T21:16:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING159445951mirror.0smithi20418.2.2-954-g82316b04trueOK59485954mirror.1smithi20418.2.2-954-g82316b04falseOKsplit-braind61657d5-3934-45cf-817b-cfff72cdb6f3up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}59445951mirror.0smithi2042024-03-23 21:16:04cluster2e9ef87d4-e149-495b-ba3d-134c4a685598down+unknownstatus not found' 2024-03-23T21:16:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-03-23T21:16:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2024-03-23T21:16:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-03-23 21:16:04' 2024-03-23T21:16:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2024-03-23T21:16:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-03-23T21:16:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-03-23T21:16:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-03-23T21:16:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-03-23T21:16:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror split-brain 2024-03-23T21:16:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:16:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:16:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-03-23T21:16:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:16:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror split-brain 2024-03-23T21:16:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:16:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-03-23T21:16:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/split-brain 2024-03-23T21:16:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/split-brain 2024-03-23T21:16:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-03-23T21:16:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-03-23T21:16:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:16:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:16:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:16:10 TEST: check if removed 'images'\''' OMAP are removed 2024-03-23T21:16:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:16:10 TEST: check if removed images' OMAP are removed 2024-03-23T21:16:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster2 2024-03-23T21:16:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:16:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:0 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:16:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:16:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-03-23T21:16:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-03-23T21:16:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:16:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:16:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-03-23T21:16:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:16:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:16:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T21:16:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:1 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-03-23T21:16:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-03-23T21:16:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-03-23T21:16:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:16:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-03-23T21:16:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:16:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:16:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-03-23T21:16:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-03-23T21:16:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:16:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-03-23T21:16:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:16:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-03-23T21:16:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=status_global 2024-03-23T21:16:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T21:16:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T21:16:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-03-23T21:16:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_ 2024-03-23T21:16:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T21:16:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:16:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_ 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T21:16:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-03-23T21:16:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_ 2024-03-23T21:16:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T21:16:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T21:16:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:16:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T21:16:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-03-23T21:16:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirror_leader 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_map 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T21:16:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-03-23T21:16:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-03-23T21:16:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-03-23T21:16:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T21:16:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-03-23T21:16:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-03-23T21:16:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-03-23T21:16:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:16:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-03-23T21:16:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T21:16:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:16:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-03-23T21:16:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T21:16:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-03-23T21:16:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-03-23T21:16:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-03-23T21:16:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T21:16:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-03-23T21:16:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-03-23T21:16:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-03-23T21:16:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:16:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-03-23T21:16:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T21:16:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-03-23T21:16:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-03-23T21:16:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T21:16:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-03-23T21:16:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-03-23T21:16:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-03-23T21:16:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T21:16:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-03-23T21:16:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=status_global 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-03-23T21:16:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-03-23T21:16:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T21:16:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-03-23T21:16:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T21:16:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_ 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:16:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-03-23T21:16:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T21:16:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_ 2024-03-23T21:16:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-03-23T21:16:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-03-23T21:16:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-03-23T21:16:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_ 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-03-23T21:16:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirror_leader 2024-03-23T21:16:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_map 2024-03-23T21:16:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-03-23T21:16:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-03-23T21:16:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-03-23T21:16:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: no blocklists' 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.TE9AI0RDtm/rbd-mirror.test.log 2024-03-23T21:16:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-03-23T21:16:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-03-23 21:16:13 TEST: no blocklists 2024-03-23T21:16:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-03-23 21:16:13 TEST: no blocklists 2024-03-23T21:16:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS='--id admin' 2024-03-23T21:16:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd blocklist ls 2024-03-23T21:16:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -q 'listed 0 entries' 2024-03-23T21:16:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS='--id admin' 2024-03-23T21:16:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd blocklist ls 2024-03-23T21:16:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -q 'listed 0 entries' 2024-03-23T21:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cleanup 0 2024-03-23T21:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code=0 2024-03-23T21:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-03-23T21:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -ne 0 ']' 2024-03-23T21:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-03-23T21:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster instance 2024-03-23T21:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:16:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-03-23T21:16:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' does not exist 2024-03-23T21:16:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:16:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-03-23T21:16:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' does not exist 2024-03-23T21:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-03-23T21:16:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' does not exist 2024-03-23T21:16:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-03-23T21:16:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-03-23T21:16:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' does not exist 2024-03-23T21:16:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-03-23T21:16:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster1 2024-03-23T21:16:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-03-23T21:16:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:16:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:16:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:16:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:0 '' 2024-03-23T21:16:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-03-23T21:16:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:16:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:16:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:16:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:0 2024-03-23T21:16:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:0 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:16:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-03-23T21:16:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:16:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:16:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=67278 2024-03-23T21:16:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 67278 ']' 2024-03-23T21:16:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 67278 2024-03-23T21:16:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:16:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:16:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:16:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=67278 '$2 == pid {print; exit 1}' 2024-03-23T21:16:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:16:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:16:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=67278 '$2 == pid {print; exit 1}' 2024-03-23T21:16:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:16:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:16:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:16:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:16:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:16:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:16:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:16:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:16:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:16:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:16:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:16:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:16:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:16:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:16:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:16:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-03-23T21:16:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:16:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:0 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:16:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:16:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-03-23T21:16:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:16:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:16:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:16:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:16:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-03-23T21:16:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:16:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-03-23T21:16:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:16:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:1 '' 2024-03-23T21:16:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-03-23T21:16:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:16:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:16:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:16:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:16:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-03-23T21:16:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:16:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:16:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=67306 2024-03-23T21:16:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 67306 ']' 2024-03-23T21:16:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 67306 2024-03-23T21:16:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:16:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:16:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:16:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=67306 '$2 == pid {print; exit 1}' 2024-03-23T21:16:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:16:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:16:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=67306 '$2 == pid {print; exit 1}' 2024-03-23T21:16:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-03-23T21:16:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:16:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:16:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:16:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:16:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:16:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:16:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-03-23T21:16:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-03-23T21:16:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-03-23T21:16:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-03-23T21:16:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:16:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:16:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-03-23T21:16:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:16:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:1 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-03-23T21:16:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-03-23T21:16:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:16:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:16:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-03-23T21:16:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:16:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-03-23T21:16:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-03-23T21:16:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster2 2024-03-23T21:16:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-03-23T21:16:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:16:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-03-23T21:16:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:16:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster2:0 '' 2024-03-23T21:16:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-03-23T21:16:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:16:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:16:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:16:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster2:0 2024-03-23T21:16:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2:0 2024-03-23T21:16:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:16:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster2 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster2 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-03-23T21:16:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-03-23T21:16:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-03-23T21:16:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-03-23T21:16:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=70974 2024-03-23T21:16:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 70974 ']' 2024-03-23T21:16:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 70974 2024-03-23T21:16:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:16:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:16:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:16:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=70974 '$2 == pid {print; exit 1}' 2024-03-23T21:16:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:16:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:16:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=70974 '$2 == pid {print; exit 1}' 2024-03-23T21:16:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-03-23T21:16:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:16:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:16:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:16:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:16:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:16:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:16:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:16:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:16:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:16:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:16:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:16:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:16:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:16:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:16:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-03-23T21:16:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-03-23T21:16:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-03-23T21:16:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:16:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:16:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-03-23T21:16:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:16:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-03-23T21:16:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster2:0 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-03-23T21:16:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-03-23T21:16:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:16:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:16:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-03-23T21:16:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-03-23T21:16:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-03-23T21:16:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-03-23T21:16:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-03-23T21:16:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-03-23T21:16:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster2:1 '' 2024-03-23T21:16:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-03-23T21:16:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-03-23T21:16:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-03-23T21:16:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-03-23T21:16:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster2:1 2024-03-23T21:16:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2:1 2024-03-23T21:16:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-03-23T21:16:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:16:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-03-23T21:16:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-03-23T21:16:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-03-23T21:16:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:16:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:16:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-03-23T21:16:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster2 2024-03-23T21:16:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster2 2024-03-23T21:16:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-03-23T21:16:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-03-23T21:16:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-03-23T21:16:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-03-23T21:16:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-03-23T21:16:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=71002 2024-03-23T21:16:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 71002 ']' 2024-03-23T21:16:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 71002 2024-03-23T21:16:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-03-23T21:16:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-03-23T21:16:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:16:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=71002 '$2 == pid {print; exit 1}' 2024-03-23T21:16:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-03-23T21:16:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-03-23T21:16:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=71002 '$2 == pid {print; exit 1}' 2024-03-23T21:16:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-03-23T21:16:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:16:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-03-23T21:16:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:16:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:16:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:16:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:16:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:16:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-03-23T21:16:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-03-23T21:16:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:16:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:16:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-03-23T21:16:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:16:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-03-23T21:16:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster2:1 2024-03-23T21:16:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-03-23T21:16:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-03-23T21:16:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-03-23T21:16:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-03-23T21:16:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-03-23T21:16:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-03-23T21:16:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-03-23T21:16:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-03-23T21:16:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z 1 ']' 2024-03-23T21:16:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test '' = /tmp/tmp.TE9AI0RDtm 2024-03-23T21:16:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -Rf /tmp/tmp.TE9AI0RDtm 2024-03-23T21:16:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-03-23T21:16:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo OK 2024-03-23T21:16:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:OK 2024-03-23T21:16:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ exit 0 2024-03-23T21:16:22.965 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-23T21:16:22.965 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-03-23T21:16:23.018 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_journal.sh'] on cluster1.client.mirror... 2024-03-23T21:16:23.018 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-03-23T21:16:23.428 DEBUG:teuthology.parallel:result is None 2024-03-23T21:16:23.429 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-03-23T21:16:23.441 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-03-23T21:16:23.441 DEBUG:teuthology.orchestra.run.smithi204:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-03-23T21:16:23.499 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-03-23T21:16:23.499 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-23T21:16:23.511 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-03-23T21:16:23.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-23T21:16:23.844 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T21:16:23.844 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-03-23T21:16:23.862 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":823,"stamp":"2024-03-23T21:16:21.918511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":128492,"kb_used_data":4812,"kb_used_omap":4,"kb_used_meta":123643,"kb_avail":282987028,"statfs":{"total":289910292480,"available":289778716672,"internally_reserved":0,"allocated":4927488,"data_stored":3027517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":126610771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-61571,"num_read_kb":-259091,"num_write":-8459,"num_write_kb":-24471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-32768,"num_keys_recovered":0,"num_objects_omap":-18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003012"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":151,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-23T21:16:11.902226+0000","last_change":"2024-03-23T20:53:52.996000+0000","last_active":"2024-03-23T21:16:11.902226+0000","last_peered":"2024-03-23T21:16:11.902226+0000","last_clean":"2024-03-23T21:16:11.902226+0000","last_became_active":"2024-03-23T20:53:52.995659+0000","last_became_peered":"2024-03-23T20:53:52.995659+0000","last_unstale":"2024-03-23T21:16:11.902226+0000","last_undegraded":"2024-03-23T21:16:11.902226+0000","last_fullsized":"2024-03-23T21:16:11.902226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:26.628717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":151,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-23T21:16:11.901844+0000","last_change":"2024-03-23T20:53:52.995652+0000","last_active":"2024-03-23T21:16:11.901844+0000","last_peered":"2024-03-23T21:16:11.901844+0000","last_clean":"2024-03-23T21:16:11.901844+0000","last_became_active":"2024-03-23T20:53:52.995094+0000","last_became_peered":"2024-03-23T20:53:52.995094+0000","last_unstale":"2024-03-23T21:16:11.901844+0000","last_undegraded":"2024-03-23T21:16:11.901844+0000","last_fullsized":"2024-03-23T21:16:11.901844+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:18.937950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":151,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-23T21:16:11.902805+0000","last_change":"2024-03-23T20:53:52.994376+0000","last_active":"2024-03-23T21:16:11.902805+0000","last_peered":"2024-03-23T21:16:11.902805+0000","last_clean":"2024-03-23T21:16:11.902805+0000","last_became_active":"2024-03-23T20:53:52.994087+0000","last_became_peered":"2024-03-23T20:53:52.994087+0000","last_unstale":"2024-03-23T21:16:11.902805+0000","last_undegraded":"2024-03-23T21:16:11.902805+0000","last_fullsized":"2024-03-23T21:16:11.902805+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:27.484370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":151,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-23T21:16:11.899193+0000","last_change":"2024-03-23T20:53:52.995843+0000","last_active":"2024-03-23T21:16:11.899193+0000","last_peered":"2024-03-23T21:16:11.899193+0000","last_clean":"2024-03-23T21:16:11.899193+0000","last_became_active":"2024-03-23T20:53:52.995018+0000","last_became_peered":"2024-03-23T20:53:52.995018+0000","last_unstale":"2024-03-23T21:16:11.899193+0000","last_undegraded":"2024-03-23T21:16:11.899193+0000","last_fullsized":"2024-03-23T21:16:11.899193+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:02.274898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":150,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-03-23T21:16:04.297443+0000","last_change":"2024-03-23T20:53:52.993885+0000","last_active":"2024-03-23T21:16:04.297443+0000","last_peered":"2024-03-23T21:16:04.297443+0000","last_clean":"2024-03-23T21:16:04.297443+0000","last_became_active":"2024-03-23T20:53:52.993597+0000","last_became_peered":"2024-03-23T20:53:52.993597+0000","last_unstale":"2024-03-23T21:16:04.297443+0000","last_undegraded":"2024-03-23T21:16:04.297443+0000","last_fullsized":"2024-03-23T21:16:04.297443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:16.901421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":151,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-23T21:16:11.370988+0000","last_change":"2024-03-23T20:53:52.994757+0000","last_active":"2024-03-23T21:16:11.370988+0000","last_peered":"2024-03-23T21:16:11.370988+0000","last_clean":"2024-03-23T21:16:11.370988+0000","last_became_active":"2024-03-23T20:53:52.994351+0000","last_became_peered":"2024-03-23T20:53:52.994351+0000","last_unstale":"2024-03-23T21:16:11.370988+0000","last_undegraded":"2024-03-23T21:16:11.370988+0000","last_fullsized":"2024-03-23T21:16:11.370988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:46.671091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-23T21:16:11.369584+0000","last_change":"2024-03-23T20:53:52.993925+0000","last_active":"2024-03-23T21:16:11.369584+0000","last_peered":"2024-03-23T21:16:11.369584+0000","last_clean":"2024-03-23T21:16:11.369584+0000","last_became_active":"2024-03-23T20:53:52.993514+0000","last_became_peered":"2024-03-23T20:53:52.993514+0000","last_unstale":"2024-03-23T21:16:11.369584+0000","last_undegraded":"2024-03-23T21:16:11.369584+0000","last_fullsized":"2024-03-23T21:16:11.369584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:44:48.653382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":151,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-23T21:16:11.901675+0000","last_change":"2024-03-23T20:53:52.992887+0000","last_active":"2024-03-23T21:16:11.901675+0000","last_peered":"2024-03-23T21:16:11.901675+0000","last_clean":"2024-03-23T21:16:11.901675+0000","last_became_active":"2024-03-23T20:53:52.992576+0000","last_became_peered":"2024-03-23T20:53:52.992576+0000","last_unstale":"2024-03-23T21:16:11.901675+0000","last_undegraded":"2024-03-23T21:16:11.901675+0000","last_fullsized":"2024-03-23T21:16:11.901675+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:37:04.022275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":282,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-03-23T21:16:11.899798+0000","last_change":"2024-03-23T20:53:48.968024+0000","last_active":"2024-03-23T21:16:11.899798+0000","last_peered":"2024-03-23T21:16:11.899798+0000","last_clean":"2024-03-23T21:16:11.899798+0000","last_became_active":"2024-03-23T20:53:48.967226+0000","last_became_peered":"2024-03-23T20:53:48.967226+0000","last_unstale":"2024-03-23T21:16:11.899798+0000","last_undegraded":"2024-03-23T21:16:11.899798+0000","last_fullsized":"2024-03-23T21:16:11.899798+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_clean_scrub_stamp":"2024-03-23T20:53:47.949768+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:38:34.242650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738642,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36424,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335416,"statfs":{"total":96636764160,"available":96599465984,"internally_reserved":0,"allocated":1245184,"data_stored":614229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48899999999999999,"15min":0.48699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":0.55200000000000005,"5min":0.879,"15min":0.879},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.53200000000000003},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.21199999999999999},"max":{"1min":0.68999999999999995,"5min":1.0920000000000001,"15min":3.544},"last":0.60799999999999998}]},{"osd":1,"last update":"Sat Mar 23 21:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57299999999999995,"15min":0.55500000000000005},"min":{"1min":0.252,"5min":0.252,"15min":0.20799999999999999},"max":{"1min":0.73499999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61799999999999999,"15min":0.60299999999999998},"min":{"1min":0.26700000000000002,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":0.73999999999999999,"5min":1.1990000000000001,"15min":2.7450000000000001},"last":0.57599999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738643,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1847296,"data_stored":1208710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:15:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.51100000000000001},"min":{"1min":0.29599999999999999,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":0.71799999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59999999999999998,"15min":0.54900000000000004},"min":{"1min":0.313,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":0.71299999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.56100000000000005}]},{"osd":2,"last update":"Sat Mar 23 21:15:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.71999999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33200000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.63300000000000001,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.60099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738644,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59148,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312692,"statfs":{"total":96636764160,"available":96576196608,"internally_reserved":0,"allocated":1835008,"data_stored":1204578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47499999999999998,"15min":0.47799999999999998},"min":{"1min":0.23599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.748},"last":0.435},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.60499999999999998,"15min":0.55300000000000005},"min":{"1min":0.28799999999999998,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.63800000000000001,"5min":7.6289999999999996,"15min":7.6289999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Mar 23 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51500000000000001,"15min":0.51400000000000001},"min":{"1min":0.218,"5min":0.192,"15min":0.192},"max":{"1min":0.63300000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.55400000000000005},"min":{"1min":0.25800000000000001,"5min":0.22600000000000001,"15min":0.187},"max":{"1min":0.70599999999999996,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:16:23.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-23T21:16:24.151 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T21:16:24.151 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-03-23T21:16:24.169 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":824,"stamp":"2024-03-23T21:16:23.918972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":128540,"kb_used_data":4856,"kb_used_omap":4,"kb_used_meta":123643,"kb_avail":282986980,"statfs":{"total":289910292480,"available":289778667520,"internally_reserved":0,"allocated":4972544,"data_stored":3046679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":126610771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-61571,"num_read_kb":-259091,"num_write":-8459,"num_write_kb":-24471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-32768,"num_keys_recovered":0,"num_objects_omap":-18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002367"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.316697+0000","last_change":"2024-03-23T20:53:52.996000+0000","last_active":"2024-03-23T21:16:16.316697+0000","last_peered":"2024-03-23T21:16:16.316697+0000","last_clean":"2024-03-23T21:16:16.316697+0000","last_became_active":"2024-03-23T20:53:52.995659+0000","last_became_peered":"2024-03-23T20:53:52.995659+0000","last_unstale":"2024-03-23T21:16:16.316697+0000","last_undegraded":"2024-03-23T21:16:16.316697+0000","last_fullsized":"2024-03-23T21:16:16.316697+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:26.628717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.324017+0000","last_change":"2024-03-23T20:53:52.995652+0000","last_active":"2024-03-23T21:16:16.324017+0000","last_peered":"2024-03-23T21:16:16.324017+0000","last_clean":"2024-03-23T21:16:16.324017+0000","last_became_active":"2024-03-23T20:53:52.995094+0000","last_became_peered":"2024-03-23T20:53:52.995094+0000","last_unstale":"2024-03-23T21:16:16.324017+0000","last_undegraded":"2024-03-23T21:16:16.324017+0000","last_fullsized":"2024-03-23T21:16:16.324017+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:18.937950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.319772+0000","last_change":"2024-03-23T20:53:52.994376+0000","last_active":"2024-03-23T21:16:16.319772+0000","last_peered":"2024-03-23T21:16:16.319772+0000","last_clean":"2024-03-23T21:16:16.319772+0000","last_became_active":"2024-03-23T20:53:52.994087+0000","last_became_peered":"2024-03-23T20:53:52.994087+0000","last_unstale":"2024-03-23T21:16:16.319772+0000","last_undegraded":"2024-03-23T21:16:16.319772+0000","last_fullsized":"2024-03-23T21:16:16.319772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:27.484370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.316316+0000","last_change":"2024-03-23T20:53:52.995843+0000","last_active":"2024-03-23T21:16:16.316316+0000","last_peered":"2024-03-23T21:16:16.316316+0000","last_clean":"2024-03-23T21:16:16.316316+0000","last_became_active":"2024-03-23T20:53:52.995018+0000","last_became_peered":"2024-03-23T20:53:52.995018+0000","last_unstale":"2024-03-23T21:16:16.316316+0000","last_undegraded":"2024-03-23T21:16:16.316316+0000","last_fullsized":"2024-03-23T21:16:16.316316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:02.274898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.959846+0000","last_change":"2024-03-23T20:53:52.993885+0000","last_active":"2024-03-23T21:16:17.959846+0000","last_peered":"2024-03-23T21:16:17.959846+0000","last_clean":"2024-03-23T21:16:17.959846+0000","last_became_active":"2024-03-23T20:53:52.993597+0000","last_became_peered":"2024-03-23T20:53:52.993597+0000","last_unstale":"2024-03-23T21:16:17.959846+0000","last_undegraded":"2024-03-23T21:16:17.959846+0000","last_fullsized":"2024-03-23T21:16:17.959846+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:16.901421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.673163+0000","last_change":"2024-03-23T20:53:52.994757+0000","last_active":"2024-03-23T21:16:16.673163+0000","last_peered":"2024-03-23T21:16:16.673163+0000","last_clean":"2024-03-23T21:16:16.673163+0000","last_became_active":"2024-03-23T20:53:52.994351+0000","last_became_peered":"2024-03-23T20:53:52.994351+0000","last_unstale":"2024-03-23T21:16:16.673163+0000","last_undegraded":"2024-03-23T21:16:16.673163+0000","last_fullsized":"2024-03-23T21:16:16.673163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:46.671091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":153,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.668327+0000","last_change":"2024-03-23T20:53:52.993925+0000","last_active":"2024-03-23T21:16:16.668327+0000","last_peered":"2024-03-23T21:16:16.668327+0000","last_clean":"2024-03-23T21:16:16.668327+0000","last_became_active":"2024-03-23T20:53:52.993514+0000","last_became_peered":"2024-03-23T20:53:52.993514+0000","last_unstale":"2024-03-23T21:16:16.668327+0000","last_undegraded":"2024-03-23T21:16:16.668327+0000","last_fullsized":"2024-03-23T21:16:16.668327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:44:48.653382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.317023+0000","last_change":"2024-03-23T20:53:52.992887+0000","last_active":"2024-03-23T21:16:16.317023+0000","last_peered":"2024-03-23T21:16:16.317023+0000","last_clean":"2024-03-23T21:16:16.317023+0000","last_became_active":"2024-03-23T20:53:52.992576+0000","last_became_peered":"2024-03-23T20:53:52.992576+0000","last_unstale":"2024-03-23T21:16:16.317023+0000","last_undegraded":"2024-03-23T21:16:16.317023+0000","last_fullsized":"2024-03-23T21:16:16.317023+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:37:04.022275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":283,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.312709+0000","last_change":"2024-03-23T20:53:48.968024+0000","last_active":"2024-03-23T21:16:16.312709+0000","last_peered":"2024-03-23T21:16:16.312709+0000","last_clean":"2024-03-23T21:16:16.312709+0000","last_became_active":"2024-03-23T20:53:48.967226+0000","last_became_peered":"2024-03-23T20:53:48.967226+0000","last_unstale":"2024-03-23T21:16:16.312709+0000","last_undegraded":"2024-03-23T21:16:16.312709+0000","last_fullsized":"2024-03-23T21:16:16.312709+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_clean_scrub_stamp":"2024-03-23T20:53:47.949768+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:38:34.242650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738643,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36444,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335396,"statfs":{"total":96636764160,"available":96599445504,"internally_reserved":0,"allocated":1261568,"data_stored":621981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48899999999999999,"15min":0.48699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":0.55200000000000005,"5min":0.879,"15min":0.879},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.53200000000000003},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.21199999999999999},"max":{"1min":0.68999999999999995,"5min":1.0920000000000001,"15min":3.544},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Mar 23 21:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57299999999999995,"15min":0.55500000000000005},"min":{"1min":0.252,"5min":0.252,"15min":0.20799999999999999},"max":{"1min":0.73499999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.495},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61799999999999999,"15min":0.60299999999999998},"min":{"1min":0.26700000000000002,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":0.73999999999999999,"5min":1.1990000000000001,"15min":2.7450000000000001},"last":0.41899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738644,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:15:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.51100000000000001},"min":{"1min":0.29599999999999999,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":0.71799999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59999999999999998,"15min":0.54900000000000004},"min":{"1min":0.313,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":0.71299999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Mar 23 21:15:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.71999999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33200000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.63300000000000001,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738645,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59168,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312672,"statfs":{"total":96636764160,"available":96576176128,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47499999999999998,"15min":0.47799999999999998},"min":{"1min":0.23599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.748},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.60499999999999998,"15min":0.55300000000000005},"min":{"1min":0.28799999999999998,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.63800000000000001,"5min":7.6289999999999996,"15min":7.6289999999999996},"last":0.29099999999999998}]},{"osd":2,"last update":"Sat Mar 23 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51500000000000001,"15min":0.51400000000000001},"min":{"1min":0.218,"5min":0.192,"15min":0.192},"max":{"1min":0.63300000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.55400000000000005},"min":{"1min":0.25800000000000001,"5min":0.22600000000000001,"15min":0.187},"max":{"1min":0.70599999999999996,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.38900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:16:24.170 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-03-23T21:16:24.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-23T21:16:24.464 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T21:16:24.464 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-03-23T21:16:24.482 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":824,"stamp":"2024-03-23T21:16:23.918972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":128540,"kb_used_data":4856,"kb_used_omap":4,"kb_used_meta":123643,"kb_avail":282986980,"statfs":{"total":289910292480,"available":289778667520,"internally_reserved":0,"allocated":4972544,"data_stored":3046679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":126610771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-61571,"num_read_kb":-259091,"num_write":-8459,"num_write_kb":-24471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-32768,"num_keys_recovered":0,"num_objects_omap":-18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002367"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.316697+0000","last_change":"2024-03-23T20:53:52.996000+0000","last_active":"2024-03-23T21:16:16.316697+0000","last_peered":"2024-03-23T21:16:16.316697+0000","last_clean":"2024-03-23T21:16:16.316697+0000","last_became_active":"2024-03-23T20:53:52.995659+0000","last_became_peered":"2024-03-23T20:53:52.995659+0000","last_unstale":"2024-03-23T21:16:16.316697+0000","last_undegraded":"2024-03-23T21:16:16.316697+0000","last_fullsized":"2024-03-23T21:16:16.316697+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:26.628717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.324017+0000","last_change":"2024-03-23T20:53:52.995652+0000","last_active":"2024-03-23T21:16:16.324017+0000","last_peered":"2024-03-23T21:16:16.324017+0000","last_clean":"2024-03-23T21:16:16.324017+0000","last_became_active":"2024-03-23T20:53:52.995094+0000","last_became_peered":"2024-03-23T20:53:52.995094+0000","last_unstale":"2024-03-23T21:16:16.324017+0000","last_undegraded":"2024-03-23T21:16:16.324017+0000","last_fullsized":"2024-03-23T21:16:16.324017+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:18.937950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.319772+0000","last_change":"2024-03-23T20:53:52.994376+0000","last_active":"2024-03-23T21:16:16.319772+0000","last_peered":"2024-03-23T21:16:16.319772+0000","last_clean":"2024-03-23T21:16:16.319772+0000","last_became_active":"2024-03-23T20:53:52.994087+0000","last_became_peered":"2024-03-23T20:53:52.994087+0000","last_unstale":"2024-03-23T21:16:16.319772+0000","last_undegraded":"2024-03-23T21:16:16.319772+0000","last_fullsized":"2024-03-23T21:16:16.319772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:27.484370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.316316+0000","last_change":"2024-03-23T20:53:52.995843+0000","last_active":"2024-03-23T21:16:16.316316+0000","last_peered":"2024-03-23T21:16:16.316316+0000","last_clean":"2024-03-23T21:16:16.316316+0000","last_became_active":"2024-03-23T20:53:52.995018+0000","last_became_peered":"2024-03-23T20:53:52.995018+0000","last_unstale":"2024-03-23T21:16:16.316316+0000","last_undegraded":"2024-03-23T21:16:16.316316+0000","last_fullsized":"2024-03-23T21:16:16.316316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:02.274898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.959846+0000","last_change":"2024-03-23T20:53:52.993885+0000","last_active":"2024-03-23T21:16:17.959846+0000","last_peered":"2024-03-23T21:16:17.959846+0000","last_clean":"2024-03-23T21:16:17.959846+0000","last_became_active":"2024-03-23T20:53:52.993597+0000","last_became_peered":"2024-03-23T20:53:52.993597+0000","last_unstale":"2024-03-23T21:16:17.959846+0000","last_undegraded":"2024-03-23T21:16:17.959846+0000","last_fullsized":"2024-03-23T21:16:17.959846+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:16.901421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.673163+0000","last_change":"2024-03-23T20:53:52.994757+0000","last_active":"2024-03-23T21:16:16.673163+0000","last_peered":"2024-03-23T21:16:16.673163+0000","last_clean":"2024-03-23T21:16:16.673163+0000","last_became_active":"2024-03-23T20:53:52.994351+0000","last_became_peered":"2024-03-23T20:53:52.994351+0000","last_unstale":"2024-03-23T21:16:16.673163+0000","last_undegraded":"2024-03-23T21:16:16.673163+0000","last_fullsized":"2024-03-23T21:16:16.673163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:46.671091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":153,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.668327+0000","last_change":"2024-03-23T20:53:52.993925+0000","last_active":"2024-03-23T21:16:16.668327+0000","last_peered":"2024-03-23T21:16:16.668327+0000","last_clean":"2024-03-23T21:16:16.668327+0000","last_became_active":"2024-03-23T20:53:52.993514+0000","last_became_peered":"2024-03-23T20:53:52.993514+0000","last_unstale":"2024-03-23T21:16:16.668327+0000","last_undegraded":"2024-03-23T21:16:16.668327+0000","last_fullsized":"2024-03-23T21:16:16.668327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:44:48.653382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.317023+0000","last_change":"2024-03-23T20:53:52.992887+0000","last_active":"2024-03-23T21:16:16.317023+0000","last_peered":"2024-03-23T21:16:16.317023+0000","last_clean":"2024-03-23T21:16:16.317023+0000","last_became_active":"2024-03-23T20:53:52.992576+0000","last_became_peered":"2024-03-23T20:53:52.992576+0000","last_unstale":"2024-03-23T21:16:16.317023+0000","last_undegraded":"2024-03-23T21:16:16.317023+0000","last_fullsized":"2024-03-23T21:16:16.317023+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:37:04.022275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":283,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.312709+0000","last_change":"2024-03-23T20:53:48.968024+0000","last_active":"2024-03-23T21:16:16.312709+0000","last_peered":"2024-03-23T21:16:16.312709+0000","last_clean":"2024-03-23T21:16:16.312709+0000","last_became_active":"2024-03-23T20:53:48.967226+0000","last_became_peered":"2024-03-23T20:53:48.967226+0000","last_unstale":"2024-03-23T21:16:16.312709+0000","last_undegraded":"2024-03-23T21:16:16.312709+0000","last_fullsized":"2024-03-23T21:16:16.312709+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_clean_scrub_stamp":"2024-03-23T20:53:47.949768+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:38:34.242650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738643,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36444,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335396,"statfs":{"total":96636764160,"available":96599445504,"internally_reserved":0,"allocated":1261568,"data_stored":621981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48899999999999999,"15min":0.48699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":0.55200000000000005,"5min":0.879,"15min":0.879},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.53200000000000003},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.21199999999999999},"max":{"1min":0.68999999999999995,"5min":1.0920000000000001,"15min":3.544},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Mar 23 21:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57299999999999995,"15min":0.55500000000000005},"min":{"1min":0.252,"5min":0.252,"15min":0.20799999999999999},"max":{"1min":0.73499999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.495},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61799999999999999,"15min":0.60299999999999998},"min":{"1min":0.26700000000000002,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":0.73999999999999999,"5min":1.1990000000000001,"15min":2.7450000000000001},"last":0.41899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738644,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:15:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.51100000000000001},"min":{"1min":0.29599999999999999,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":0.71799999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59999999999999998,"15min":0.54900000000000004},"min":{"1min":0.313,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":0.71299999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Mar 23 21:15:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.71999999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33200000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.63300000000000001,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738645,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59168,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312672,"statfs":{"total":96636764160,"available":96576176128,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47499999999999998,"15min":0.47799999999999998},"min":{"1min":0.23599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.748},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.60499999999999998,"15min":0.55300000000000005},"min":{"1min":0.28799999999999998,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.63800000000000001,"5min":7.6289999999999996,"15min":7.6289999999999996},"last":0.29099999999999998}]},{"osd":2,"last update":"Sat Mar 23 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51500000000000001,"15min":0.51400000000000001},"min":{"1min":0.218,"5min":0.192,"15min":0.192},"max":{"1min":0.63300000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.55400000000000005},"min":{"1min":0.25800000000000001,"5min":0.22600000000000001,"15min":0.187},"max":{"1min":0.70599999999999996,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.38900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:16:24.483 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-03-23T21:16:24.766 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T21:16:24.766 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":147,"fsid":"d781caf7-4325-48b0-8e95-908c0fb74304","created":"2024-03-23T20:53:35.958816+0000","modified":"2024-03-23T21:16:18.306081+0000","last_up_change":"2024-03-23T20:53:45.935300+0000","last_in_change":"2024-03-23T20:53:36.908224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T20:53:47.483126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-23T20:53:51.400689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06c6a838-cd4d-4e46-8a3a-97ec64073385","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":24013},{"type":"v1","addr":"172.21.15.204:6801","nonce":24013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":24013},{"type":"v1","addr":"172.21.15.204:6803","nonce":24013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":24013},{"type":"v1","addr":"172.21.15.204:6807","nonce":24013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":24013},{"type":"v1","addr":"172.21.15.204:6805","nonce":24013}]},"public_addr":"172.21.15.204:6801/24013","cluster_addr":"172.21.15.204:6803/24013","heartbeat_back_addr":"172.21.15.204:6807/24013","heartbeat_front_addr":"172.21.15.204:6805/24013","state":["exists","up"]},{"osd":1,"uuid":"25daa7ad-a344-420f-93ab-040a61bf446c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":24039},{"type":"v1","addr":"172.21.15.204:6817","nonce":24039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":24039},{"type":"v1","addr":"172.21.15.204:6819","nonce":24039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":24039},{"type":"v1","addr":"172.21.15.204:6823","nonce":24039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":24039},{"type":"v1","addr":"172.21.15.204:6821","nonce":24039}]},"public_addr":"172.21.15.204:6817/24039","cluster_addr":"172.21.15.204:6819/24039","heartbeat_back_addr":"172.21.15.204:6823/24039","heartbeat_front_addr":"172.21.15.204:6821/24039","state":["exists","up"]},{"osd":2,"uuid":"caa8e5a6-7291-4e08-9335-354bbd6e892f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":24038},{"type":"v1","addr":"172.21.15.204:6809","nonce":24038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":24038},{"type":"v1","addr":"172.21.15.204:6811","nonce":24038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":24038},{"type":"v1","addr":"172.21.15.204:6815","nonce":24038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":24038},{"type":"v1","addr":"172.21.15.204:6813","nonce":24038}]},"public_addr":"172.21.15.204:6809/24038","cluster_addr":"172.21.15.204:6811/24038","heartbeat_back_addr":"172.21.15.204:6815/24038","heartbeat_front_addr":"172.21.15.204:6813/24038","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.182167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.449795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:42.391287+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T21:16:25.784 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-03-23T21:16:25.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-23T21:16:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-23T21:16:25.938 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T21:16:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-23T21:16:25.954 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-03-23T21:16:26.236 INFO:teuthology.orchestra.run.smithi204.stderr:instructed osd(s) 0 to deep-scrub 2024-03-23T21:16:26.254 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-03-23T21:16:26.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-23T21:16:26.398 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-23T21:16:26.398 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T21:16:26.398 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-23T21:16:26.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-03-23T21:16:26.681 INFO:teuthology.orchestra.run.smithi204.stderr:instructed osd(s) 1 to deep-scrub 2024-03-23T21:16:26.699 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-03-23T21:16:26.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-23T21:16:26.833 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-23T21:16:26.833 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T21:16:26.833 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-23T21:16:26.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-03-23T21:16:27.112 INFO:teuthology.orchestra.run.smithi204.stderr:instructed osd(s) 2 to deep-scrub 2024-03-23T21:16:27.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-23T21:16:27.399 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T21:16:27.399 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-03-23T21:16:27.416 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":825,"stamp":"2024-03-23T21:16:25.919480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":128540,"kb_used_data":4856,"kb_used_omap":4,"kb_used_meta":123643,"kb_avail":282986980,"statfs":{"total":289910292480,"available":289778667520,"internally_reserved":0,"allocated":4972544,"data_stored":3046679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":126610771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-7,"num_object_clones":0,"num_object_copies":-14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":0,"num_read":-4111,"num_read_kb":-11336,"num_write":-858,"num_write_kb":-2255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.598906"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.316697+0000","last_change":"2024-03-23T20:53:52.996000+0000","last_active":"2024-03-23T21:16:16.316697+0000","last_peered":"2024-03-23T21:16:16.316697+0000","last_clean":"2024-03-23T21:16:16.316697+0000","last_became_active":"2024-03-23T20:53:52.995659+0000","last_became_peered":"2024-03-23T20:53:52.995659+0000","last_unstale":"2024-03-23T21:16:16.316697+0000","last_undegraded":"2024-03-23T21:16:16.316697+0000","last_fullsized":"2024-03-23T21:16:16.316697+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:38:26.628717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.324017+0000","last_change":"2024-03-23T20:53:52.995652+0000","last_active":"2024-03-23T21:16:16.324017+0000","last_peered":"2024-03-23T21:16:16.324017+0000","last_clean":"2024-03-23T21:16:16.324017+0000","last_became_active":"2024-03-23T20:53:52.995094+0000","last_became_peered":"2024-03-23T20:53:52.995094+0000","last_unstale":"2024-03-23T21:16:16.324017+0000","last_undegraded":"2024-03-23T21:16:16.324017+0000","last_fullsized":"2024-03-23T21:16:16.324017+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:18.937950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.319772+0000","last_change":"2024-03-23T20:53:52.994376+0000","last_active":"2024-03-23T21:16:16.319772+0000","last_peered":"2024-03-23T21:16:16.319772+0000","last_clean":"2024-03-23T21:16:16.319772+0000","last_became_active":"2024-03-23T20:53:52.994087+0000","last_became_peered":"2024-03-23T20:53:52.994087+0000","last_unstale":"2024-03-23T21:16:16.319772+0000","last_undegraded":"2024-03-23T21:16:16.319772+0000","last_fullsized":"2024-03-23T21:16:16.319772+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:27.484370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.316316+0000","last_change":"2024-03-23T20:53:52.995843+0000","last_active":"2024-03-23T21:16:16.316316+0000","last_peered":"2024-03-23T21:16:16.316316+0000","last_clean":"2024-03-23T21:16:16.316316+0000","last_became_active":"2024-03-23T20:53:52.995018+0000","last_became_peered":"2024-03-23T20:53:52.995018+0000","last_unstale":"2024-03-23T21:16:16.316316+0000","last_undegraded":"2024-03-23T21:16:16.316316+0000","last_fullsized":"2024-03-23T21:16:16.316316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:02.274898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.959846+0000","last_change":"2024-03-23T20:53:52.993885+0000","last_active":"2024-03-23T21:16:17.959846+0000","last_peered":"2024-03-23T21:16:17.959846+0000","last_clean":"2024-03-23T21:16:17.959846+0000","last_became_active":"2024-03-23T20:53:52.993597+0000","last_became_peered":"2024-03-23T20:53:52.993597+0000","last_unstale":"2024-03-23T21:16:17.959846+0000","last_undegraded":"2024-03-23T21:16:17.959846+0000","last_fullsized":"2024-03-23T21:16:17.959846+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:16.901421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.673163+0000","last_change":"2024-03-23T20:53:52.994757+0000","last_active":"2024-03-23T21:16:16.673163+0000","last_peered":"2024-03-23T21:16:16.673163+0000","last_clean":"2024-03-23T21:16:16.673163+0000","last_became_active":"2024-03-23T20:53:52.994351+0000","last_became_peered":"2024-03-23T20:53:52.994351+0000","last_unstale":"2024-03-23T21:16:16.673163+0000","last_undegraded":"2024-03-23T21:16:16.673163+0000","last_fullsized":"2024-03-23T21:16:16.673163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:40:46.671091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":153,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.668327+0000","last_change":"2024-03-23T20:53:52.993925+0000","last_active":"2024-03-23T21:16:16.668327+0000","last_peered":"2024-03-23T21:16:16.668327+0000","last_clean":"2024-03-23T21:16:16.668327+0000","last_became_active":"2024-03-23T20:53:52.993514+0000","last_became_peered":"2024-03-23T20:53:52.993514+0000","last_unstale":"2024-03-23T21:16:16.668327+0000","last_undegraded":"2024-03-23T21:16:16.668327+0000","last_fullsized":"2024-03-23T21:16:16.668327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:44:48.653382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.317023+0000","last_change":"2024-03-23T20:53:52.992887+0000","last_active":"2024-03-23T21:16:16.317023+0000","last_peered":"2024-03-23T21:16:16.317023+0000","last_clean":"2024-03-23T21:16:16.317023+0000","last_became_active":"2024-03-23T20:53:52.992576+0000","last_became_peered":"2024-03-23T20:53:52.992576+0000","last_unstale":"2024-03-23T21:16:16.317023+0000","last_undegraded":"2024-03-23T21:16:16.317023+0000","last_fullsized":"2024-03-23T21:16:16.317023+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:37:04.022275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":283,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-23T21:16:16.312709+0000","last_change":"2024-03-23T20:53:48.968024+0000","last_active":"2024-03-23T21:16:16.312709+0000","last_peered":"2024-03-23T21:16:16.312709+0000","last_clean":"2024-03-23T21:16:16.312709+0000","last_became_active":"2024-03-23T20:53:48.967226+0000","last_became_peered":"2024-03-23T20:53:48.967226+0000","last_unstale":"2024-03-23T21:16:16.312709+0000","last_undegraded":"2024-03-23T21:16:16.312709+0000","last_fullsized":"2024-03-23T21:16:16.312709+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:47.949768+0000","last_clean_scrub_stamp":"2024-03-23T20:53:47.949768+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:38:34.242650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738643,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36444,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335396,"statfs":{"total":96636764160,"available":96599445504,"internally_reserved":0,"allocated":1261568,"data_stored":621981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48899999999999999,"15min":0.48699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.185},"max":{"1min":0.55200000000000005,"5min":0.879,"15min":0.879},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.53200000000000003},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.21199999999999999},"max":{"1min":0.68999999999999995,"5min":1.0920000000000001,"15min":3.544},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Mar 23 21:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57299999999999995,"15min":0.55500000000000005},"min":{"1min":0.252,"5min":0.252,"15min":0.20799999999999999},"max":{"1min":0.73499999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.495},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61799999999999999,"15min":0.60299999999999998},"min":{"1min":0.26700000000000002,"5min":0.26600000000000001,"15min":0.23400000000000001},"max":{"1min":0.73999999999999999,"5min":1.1990000000000001,"15min":2.7450000000000001},"last":0.41899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738644,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:15:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.51100000000000001},"min":{"1min":0.29599999999999999,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":0.71799999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59999999999999998,"15min":0.54900000000000004},"min":{"1min":0.313,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":0.71299999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Mar 23 21:15:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.71999999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33200000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.63300000000000001,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.44700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738645,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59168,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312672,"statfs":{"total":96636764160,"available":96576176128,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47499999999999998,"15min":0.47799999999999998},"min":{"1min":0.23599999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.748},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.60499999999999998,"15min":0.55300000000000005},"min":{"1min":0.28799999999999998,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.63800000000000001,"5min":7.6289999999999996,"15min":7.6289999999999996},"last":0.29099999999999998}]},{"osd":2,"last update":"Sat Mar 23 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51500000000000001,"15min":0.51400000000000001},"min":{"1min":0.218,"5min":0.192,"15min":0.192},"max":{"1min":0.63300000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58099999999999996,"15min":0.55400000000000005},"min":{"1min":0.25800000000000001,"5min":0.22600000000000001,"15min":0.187},"max":{"1min":0.70599999999999996,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.38900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:16:27.420 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.420 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.420 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.420 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.420 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.421 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.421 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.421 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.421 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-23T20:53:47.949768+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=47, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:27.421 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-23T21:16:47.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-23T21:16:47.721 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T21:16:47.721 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-03-23T21:16:47.741 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":835,"stamp":"2024-03-23T21:16:45.924439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":128528,"kb_used_data":4856,"kb_used_omap":4,"kb_used_meta":123643,"kb_avail":282986992,"statfs":{"total":289910292480,"available":289778679808,"internally_reserved":0,"allocated":4972544,"data_stored":3046679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":126610771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003087"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":153,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:20.298291+0000","last_change":"2024-03-23T20:53:52.996000+0000","last_active":"2024-03-23T21:16:20.298291+0000","last_peered":"2024-03-23T21:16:20.298291+0000","last_clean":"2024-03-23T21:16:20.298291+0000","last_became_active":"2024-03-23T20:53:52.995659+0000","last_became_peered":"2024-03-23T20:53:52.995659+0000","last_unstale":"2024-03-23T21:16:20.298291+0000","last_undegraded":"2024-03-23T21:16:20.298291+0000","last_fullsized":"2024-03-23T21:16:20.298291+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":153,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:20.307889+0000","last_change":"2024-03-23T20:53:52.995652+0000","last_active":"2024-03-23T21:16:20.307889+0000","last_peered":"2024-03-23T21:16:20.307889+0000","last_clean":"2024-03-23T21:16:20.307889+0000","last_became_active":"2024-03-23T20:53:52.995094+0000","last_became_peered":"2024-03-23T20:53:52.995094+0000","last_unstale":"2024-03-23T21:16:20.307889+0000","last_undegraded":"2024-03-23T21:16:20.307889+0000","last_fullsized":"2024-03-23T21:16:20.307889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:20.303827+0000","last_change":"2024-03-23T20:53:52.994376+0000","last_active":"2024-03-23T21:16:20.303827+0000","last_peered":"2024-03-23T21:16:20.303827+0000","last_clean":"2024-03-23T21:16:20.303827+0000","last_became_active":"2024-03-23T20:53:52.994087+0000","last_became_peered":"2024-03-23T20:53:52.994087+0000","last_unstale":"2024-03-23T21:16:20.303827+0000","last_undegraded":"2024-03-23T21:16:20.303827+0000","last_fullsized":"2024-03-23T21:16:20.303827+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:51.972326+0000","last_clean_scrub_stamp":"2024-03-23T20:53:51.972326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":164,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:31.261155+0000","last_change":"2024-03-23T21:16:31.261386+0000","last_active":"2024-03-23T21:16:31.261155+0000","last_peered":"2024-03-23T21:16:31.261155+0000","last_clean":"2024-03-23T21:16:31.261155+0000","last_became_active":"2024-03-23T20:53:52.995018+0000","last_became_peered":"2024-03-23T20:53:52.995018+0000","last_unstale":"2024-03-23T21:16:31.261155+0000","last_undegraded":"2024-03-23T21:16:31.261155+0000","last_fullsized":"2024-03-23T21:16:31.261155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:31.261108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:31.261108+0000","last_clean_scrub_stamp":"2024-03-23T21:16:31.261108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:24.977175+0000","scrub_duration":0.002918997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":164,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:26.843233+0000","last_change":"2024-03-23T21:16:26.843329+0000","last_active":"2024-03-23T21:16:26.843233+0000","last_peered":"2024-03-23T21:16:26.843233+0000","last_clean":"2024-03-23T21:16:26.843233+0000","last_became_active":"2024-03-23T20:53:52.993597+0000","last_became_peered":"2024-03-23T20:53:52.993597+0000","last_unstale":"2024-03-23T21:16:26.843233+0000","last_undegraded":"2024-03-23T21:16:26.843233+0000","last_fullsized":"2024-03-23T21:16:26.843233+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:26.843213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:26.843213+0000","last_clean_scrub_stamp":"2024-03-23T21:16:26.843213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:50.349798+0000","scrub_duration":0.001766933,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":164,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:31.109057+0000","last_change":"2024-03-23T21:16:31.109295+0000","last_active":"2024-03-23T21:16:31.109057+0000","last_peered":"2024-03-23T21:16:31.109057+0000","last_clean":"2024-03-23T21:16:31.109057+0000","last_became_active":"2024-03-23T20:53:52.994351+0000","last_became_peered":"2024-03-23T20:53:52.994351+0000","last_unstale":"2024-03-23T21:16:31.109057+0000","last_undegraded":"2024-03-23T21:16:31.109057+0000","last_fullsized":"2024-03-23T21:16:31.109057+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:31.109007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:31.109007+0000","last_clean_scrub_stamp":"2024-03-23T21:16:31.109007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:11.559876+0000","scrub_duration":0.0050266419999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:28.144735+0000","last_change":"2024-03-23T21:16:28.144870+0000","last_active":"2024-03-23T21:16:28.144735+0000","last_peered":"2024-03-23T21:16:28.144735+0000","last_clean":"2024-03-23T21:16:28.144735+0000","last_became_active":"2024-03-23T20:53:52.993514+0000","last_became_peered":"2024-03-23T20:53:52.993514+0000","last_unstale":"2024-03-23T21:16:28.144735+0000","last_undegraded":"2024-03-23T21:16:28.144735+0000","last_fullsized":"2024-03-23T21:16:28.144735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:28.144709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:28.144709+0000","last_clean_scrub_stamp":"2024-03-23T21:16:28.144709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:51.026695+0000","scrub_duration":0.0043832840000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":164,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:40.333926+0000","last_change":"2024-03-23T21:16:40.334155+0000","last_active":"2024-03-23T21:16:40.333926+0000","last_peered":"2024-03-23T21:16:40.333926+0000","last_clean":"2024-03-23T21:16:40.333926+0000","last_became_active":"2024-03-23T20:53:52.992576+0000","last_became_peered":"2024-03-23T20:53:52.992576+0000","last_unstale":"2024-03-23T21:16:40.333926+0000","last_undegraded":"2024-03-23T21:16:40.333926+0000","last_fullsized":"2024-03-23T21:16:40.333926+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:40.333875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:40.333875+0000","last_clean_scrub_stamp":"2024-03-23T21:16:40.333875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:19.141686+0000","scrub_duration":0.006285021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"147'99","reported_seq":297,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:27.112231+0000","last_change":"2024-03-23T21:16:27.112360+0000","last_active":"2024-03-23T21:16:27.112231+0000","last_peered":"2024-03-23T21:16:27.112231+0000","last_clean":"2024-03-23T21:16:27.112231+0000","last_became_active":"2024-03-23T20:53:48.967226+0000","last_became_peered":"2024-03-23T20:53:48.967226+0000","last_unstale":"2024-03-23T21:16:27.112231+0000","last_undegraded":"2024-03-23T21:16:27.112231+0000","last_fullsized":"2024-03-23T21:16:27.112231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'99","last_scrub_stamp":"2024-03-23T21:16:27.112195+0000","last_deep_scrub":"147'99","last_deep_scrub_stamp":"2024-03-23T21:16:27.112195+0000","last_clean_scrub_stamp":"2024-03-23T21:16:27.112195+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:40:58.220076+0000","scrub_duration":0.012388758999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738647,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36440,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335400,"statfs":{"total":96636764160,"available":96599449600,"internally_reserved":0,"allocated":1261568,"data_stored":621981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.74099999999999999,"5min":0.879,"15min":0.879},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.66500000000000004,"5min":1.0920000000000001,"15min":3.544},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Mar 23 21:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.68300000000000005,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.78600000000000003,"5min":1.1990000000000001,"15min":2.7450000000000001},"last":0.58199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738648,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53300000000000003,"15min":0.51100000000000001},"min":{"1min":0.317,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":0.72399999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59499999999999997,"15min":0.55400000000000005},"min":{"1min":0.40899999999999997,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":0.64600000000000002,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.61899999999999999}]},{"osd":2,"last update":"Sat Mar 23 21:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.59399999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55000000000000004},"min":{"1min":0.44700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.68100000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738649,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59164,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312676,"statfs":{"total":96636764160,"available":96576180224,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.23999999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.61899999999999999,"5min":0.69299999999999995,"15min":0.748},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.59499999999999997,"15min":0.55200000000000005},"min":{"1min":0.29099999999999998,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.628,"5min":7.6289999999999996,"15min":7.6289999999999996},"last":0.68200000000000005}]},{"osd":2,"last update":"Sat Mar 23 21:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.28899999999999998,"5min":0.192,"15min":0.192},"max":{"1min":0.65100000000000002,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57599999999999996,"15min":0.55700000000000005},"min":{"1min":0.371,"5min":0.22600000000000001,"15min":0.187},"max":{"1min":0.73799999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.72599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:16:47.742 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:47.742 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:47.743 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-23T20:53:51.972326+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=51, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=16, tm_sec=24, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:16:47.743 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-23T21:17:07.743 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-03-23T21:17:08.051 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-23T21:17:08.051 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-03-23T21:17:08.069 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":846,"stamp":"2024-03-23T21:17:07.929926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":128528,"kb_used_data":4856,"kb_used_omap":4,"kb_used_meta":123643,"kb_avail":282986992,"statfs":{"total":289910292480,"available":289778679808,"internally_reserved":0,"allocated":4972544,"data_stored":3046679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":126610771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":164,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:56.222672+0000","last_change":"2024-03-23T21:16:56.223001+0000","last_active":"2024-03-23T21:16:56.222672+0000","last_peered":"2024-03-23T21:16:56.222672+0000","last_clean":"2024-03-23T21:16:56.222672+0000","last_became_active":"2024-03-23T20:53:52.995659+0000","last_became_peered":"2024-03-23T20:53:52.995659+0000","last_unstale":"2024-03-23T21:16:56.222672+0000","last_undegraded":"2024-03-23T21:16:56.222672+0000","last_fullsized":"2024-03-23T21:16:56.222672+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:56.222570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:56.222570+0000","last_clean_scrub_stamp":"2024-03-23T21:16:56.222570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:15:54.233525+0000","scrub_duration":0.0069559620000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":164,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:46.336093+0000","last_change":"2024-03-23T21:16:46.336343+0000","last_active":"2024-03-23T21:16:46.336093+0000","last_peered":"2024-03-23T21:16:46.336093+0000","last_clean":"2024-03-23T21:16:46.336093+0000","last_became_active":"2024-03-23T20:53:52.995094+0000","last_became_peered":"2024-03-23T20:53:52.995094+0000","last_unstale":"2024-03-23T21:16:46.336093+0000","last_undegraded":"2024-03-23T21:16:46.336093+0000","last_fullsized":"2024-03-23T21:16:46.336093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:46.336041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:46.336041+0000","last_clean_scrub_stamp":"2024-03-23T21:16:46.336041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:18:43.877307+0000","scrub_duration":0.0042006309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":165,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:50.276433+0000","last_change":"2024-03-23T21:16:50.276433+0000","last_active":"2024-03-23T21:16:50.276433+0000","last_peered":"2024-03-23T21:16:50.276433+0000","last_clean":"2024-03-23T21:16:50.276433+0000","last_became_active":"2024-03-23T20:53:52.994087+0000","last_became_peered":"2024-03-23T20:53:52.994087+0000","last_unstale":"2024-03-23T21:16:50.276433+0000","last_undegraded":"2024-03-23T21:16:50.276433+0000","last_fullsized":"2024-03-23T21:16:50.276433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:50.276106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:50.276106+0000","last_clean_scrub_stamp":"2024-03-23T21:16:50.276106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:18.585460+0000","scrub_duration":0.0064407270000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":165,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:31.261386+0000","last_change":"2024-03-23T21:16:31.261386+0000","last_active":"2024-03-23T21:16:31.261386+0000","last_peered":"2024-03-23T21:16:31.261386+0000","last_clean":"2024-03-23T21:16:31.261386+0000","last_became_active":"2024-03-23T20:53:52.995018+0000","last_became_peered":"2024-03-23T20:53:52.995018+0000","last_unstale":"2024-03-23T21:16:31.261386+0000","last_undegraded":"2024-03-23T21:16:31.261386+0000","last_fullsized":"2024-03-23T21:16:31.261386+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:31.261108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:31.261108+0000","last_clean_scrub_stamp":"2024-03-23T21:16:31.261108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:06:24.977175+0000","scrub_duration":0.002918997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":165,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:26.843329+0000","last_change":"2024-03-23T21:16:26.843329+0000","last_active":"2024-03-23T21:16:26.843329+0000","last_peered":"2024-03-23T21:16:26.843329+0000","last_clean":"2024-03-23T21:16:26.843329+0000","last_became_active":"2024-03-23T20:53:52.993597+0000","last_became_peered":"2024-03-23T20:53:52.993597+0000","last_unstale":"2024-03-23T21:16:26.843329+0000","last_undegraded":"2024-03-23T21:16:26.843329+0000","last_fullsized":"2024-03-23T21:16:26.843329+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:26.843213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:26.843213+0000","last_clean_scrub_stamp":"2024-03-23T21:16:26.843213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:47:50.349798+0000","scrub_duration":0.001766933,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":165,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:31.109295+0000","last_change":"2024-03-23T21:16:31.109295+0000","last_active":"2024-03-23T21:16:31.109295+0000","last_peered":"2024-03-23T21:16:31.109295+0000","last_clean":"2024-03-23T21:16:31.109295+0000","last_became_active":"2024-03-23T20:53:52.994351+0000","last_became_peered":"2024-03-23T20:53:52.994351+0000","last_unstale":"2024-03-23T21:16:31.109295+0000","last_undegraded":"2024-03-23T21:16:31.109295+0000","last_fullsized":"2024-03-23T21:16:31.109295+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:31.109007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:31.109007+0000","last_clean_scrub_stamp":"2024-03-23T21:16:31.109007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:11.559876+0000","scrub_duration":0.0050266419999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:28.144735+0000","last_change":"2024-03-23T21:16:28.144870+0000","last_active":"2024-03-23T21:16:28.144735+0000","last_peered":"2024-03-23T21:16:28.144735+0000","last_clean":"2024-03-23T21:16:28.144735+0000","last_became_active":"2024-03-23T20:53:52.993514+0000","last_became_peered":"2024-03-23T20:53:52.993514+0000","last_unstale":"2024-03-23T21:16:28.144735+0000","last_undegraded":"2024-03-23T21:16:28.144735+0000","last_fullsized":"2024-03-23T21:16:28.144735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:28.144709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:28.144709+0000","last_clean_scrub_stamp":"2024-03-23T21:16:28.144709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:51.026695+0000","scrub_duration":0.0043832840000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":165,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:40.334155+0000","last_change":"2024-03-23T21:16:40.334155+0000","last_active":"2024-03-23T21:16:40.334155+0000","last_peered":"2024-03-23T21:16:40.334155+0000","last_clean":"2024-03-23T21:16:40.334155+0000","last_became_active":"2024-03-23T20:53:52.992576+0000","last_became_peered":"2024-03-23T20:53:52.992576+0000","last_unstale":"2024-03-23T21:16:40.334155+0000","last_undegraded":"2024-03-23T21:16:40.334155+0000","last_fullsized":"2024-03-23T21:16:40.334155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:16:40.333875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:16:40.333875+0000","last_clean_scrub_stamp":"2024-03-23T21:16:40.333875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:15:19.141686+0000","scrub_duration":0.006285021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"147'99","reported_seq":297,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-23T21:16:27.112231+0000","last_change":"2024-03-23T21:16:27.112360+0000","last_active":"2024-03-23T21:16:27.112231+0000","last_peered":"2024-03-23T21:16:27.112231+0000","last_clean":"2024-03-23T21:16:27.112231+0000","last_became_active":"2024-03-23T20:53:48.967226+0000","last_became_peered":"2024-03-23T20:53:48.967226+0000","last_unstale":"2024-03-23T21:16:27.112231+0000","last_undegraded":"2024-03-23T21:16:27.112231+0000","last_fullsized":"2024-03-23T21:16:27.112231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'99","last_scrub_stamp":"2024-03-23T21:16:27.112195+0000","last_deep_scrub":"147'99","last_deep_scrub_stamp":"2024-03-23T21:16:27.112195+0000","last_clean_scrub_stamp":"2024-03-23T21:16:27.112195+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:40:58.220076+0000","scrub_duration":0.012388758999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738652,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36440,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335400,"statfs":{"total":96636764160,"available":96599449600,"internally_reserved":0,"allocated":1261568,"data_stored":621981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49299999999999999,"15min":0.48599999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.74099999999999999,"5min":0.879,"15min":0.879},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.66500000000000004,"5min":1.0920000000000001,"15min":3.544},"last":0.67900000000000005}]},{"osd":1,"last update":"Sat Mar 23 21:16:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.68300000000000005,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.748},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.78600000000000003,"5min":1.1990000000000001,"15min":2.7450000000000001},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738653,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53300000000000003,"15min":0.51100000000000001},"min":{"1min":0.317,"5min":0.216,"15min":0.19500000000000001},"max":{"1min":0.72399999999999998,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59499999999999997,"15min":0.55400000000000005},"min":{"1min":0.40899999999999997,"5min":0.22800000000000001,"15min":0.22600000000000001},"max":{"1min":0.64600000000000002,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Mar 23 21:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.59399999999999997,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.55000000000000004},"min":{"1min":0.44700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.68100000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738654,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59164,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312676,"statfs":{"total":96636764160,"available":96576180224,"internally_reserved":0,"allocated":1855488,"data_stored":1212349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.23999999999999999,"5min":0.155,"15min":0.155},"max":{"1min":0.61899999999999999,"5min":0.69299999999999995,"15min":0.748},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.59499999999999997,"15min":0.55200000000000005},"min":{"1min":0.29099999999999998,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.628,"5min":7.6289999999999996,"15min":7.6289999999999996},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Mar 23 21:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.28899999999999998,"5min":0.192,"15min":0.192},"max":{"1min":0.65100000000000002,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57599999999999996,"15min":0.55700000000000005},"min":{"1min":0.371,"5min":0.22600000000000001,"15min":0.187},"max":{"1min":0.73799999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:17:08.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-03-23T21:17:08.388 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-23T21:17:08.389 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-23T21:17:08.389 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-03-23T21:17:08.389 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:08.458 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-03-23T21:17:08.459 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-03-23T21:17:08.459 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:11.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-03-23T21:17:14.560 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-03-23T21:17:14.561 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-03-23T21:17:14.561 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:14.623 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-03-23T21:17:14.623 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-23T21:17:14.623 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-03-23T21:17:14.623 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:14.651 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-03-23T21:17:14.651 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-23T21:17:14.651 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-03-23T21:17:14.652 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:14.678 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-03-23T21:17:14.678 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-23T21:17:14.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-03-23T21:17:14.735 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T21:17:14.735 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-03-23T21:17:15.011 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T21:17:15.012 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-03-23T21:17:15.217 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T21:17:15.217 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-03-23T21:17:15.377 INFO:tasks.ceph:Archiving mon data... 2024-03-23T21:17:15.379 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/data/cluster2.mon.a.tgz 2024-03-23T21:17:15.379 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-23T21:17:15.384 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.eLhYiVySwM 2024-03-23T21:17:15.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.eLhYiVySwM 2024-03-23T21:17:15.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.eLhYiVySwM 2024-03-23T21:17:15.580 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.eLhYiVySwM is 328KB 2024-03-23T21:17:15.637 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.eLhYiVySwM 2024-03-23T21:17:15.642 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-23T21:17:15.643 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T21:17:15.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T21:17:15.701 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-23T21:17:15.710 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-03-23T21:17:15.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-23T21:17:16.006 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T21:17:16.006 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-23T21:17:16.024 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":872,"stamp":"2024-03-23T21:17:14.345428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":136828,"kb_used_data":4948,"kb_used_omap":4,"kb_used_meta":131835,"kb_avail":282978692,"statfs":{"total":289910292480,"available":289770180608,"internally_reserved":0,"allocated":5066752,"data_stored":3122206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":134999378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003380"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.478953+0000","last_change":"2024-03-23T20:53:16.260778+0000","last_active":"2024-03-23T21:16:17.478953+0000","last_peered":"2024-03-23T21:16:17.478953+0000","last_clean":"2024-03-23T21:16:17.478953+0000","last_became_active":"2024-03-23T20:53:16.260429+0000","last_became_peered":"2024-03-23T20:53:16.260429+0000","last_unstale":"2024-03-23T21:16:17.478953+0000","last_undegraded":"2024-03-23T21:16:17.478953+0000","last_fullsized":"2024-03-23T21:16:17.478953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:08:21.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.486080+0000","last_change":"2024-03-23T20:53:16.257714+0000","last_active":"2024-03-23T21:16:17.486080+0000","last_peered":"2024-03-23T21:16:17.486080+0000","last_clean":"2024-03-23T21:16:17.486080+0000","last_became_active":"2024-03-23T20:53:16.257287+0000","last_became_peered":"2024-03-23T20:53:16.257287+0000","last_unstale":"2024-03-23T21:16:17.486080+0000","last_undegraded":"2024-03-23T21:16:17.486080+0000","last_fullsized":"2024-03-23T21:16:17.486080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:25.116796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.483398+0000","last_change":"2024-03-23T20:53:16.260048+0000","last_active":"2024-03-23T21:16:17.483398+0000","last_peered":"2024-03-23T21:16:17.483398+0000","last_clean":"2024-03-23T21:16:17.483398+0000","last_became_active":"2024-03-23T20:53:16.259683+0000","last_became_peered":"2024-03-23T20:53:16.259683+0000","last_unstale":"2024-03-23T21:16:17.483398+0000","last_undegraded":"2024-03-23T21:16:17.483398+0000","last_fullsized":"2024-03-23T21:16:17.483398+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:39:53.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.465220+0000","last_change":"2024-03-23T20:53:16.260803+0000","last_active":"2024-03-23T21:16:15.465220+0000","last_peered":"2024-03-23T21:16:15.465220+0000","last_clean":"2024-03-23T21:16:15.465220+0000","last_became_active":"2024-03-23T20:53:16.260458+0000","last_became_peered":"2024-03-23T20:53:16.260458+0000","last_unstale":"2024-03-23T21:16:15.465220+0000","last_undegraded":"2024-03-23T21:16:15.465220+0000","last_fullsized":"2024-03-23T21:16:15.465220+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:25:51.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":156,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.656552+0000","last_change":"2024-03-23T20:53:16.258624+0000","last_active":"2024-03-23T21:16:15.656552+0000","last_peered":"2024-03-23T21:16:15.656552+0000","last_clean":"2024-03-23T21:16:15.656552+0000","last_became_active":"2024-03-23T20:53:16.257825+0000","last_became_peered":"2024-03-23T20:53:16.257825+0000","last_unstale":"2024-03-23T21:16:15.656552+0000","last_undegraded":"2024-03-23T21:16:15.656552+0000","last_fullsized":"2024-03-23T21:16:15.656552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:55.573156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.988259+0000","last_change":"2024-03-23T20:53:16.257583+0000","last_active":"2024-03-23T21:16:15.988259+0000","last_peered":"2024-03-23T21:16:15.988259+0000","last_clean":"2024-03-23T21:16:15.988259+0000","last_became_active":"2024-03-23T20:53:16.257206+0000","last_became_peered":"2024-03-23T20:53:16.257206+0000","last_unstale":"2024-03-23T21:16:15.988259+0000","last_undegraded":"2024-03-23T21:16:15.988259+0000","last_fullsized":"2024-03-23T21:16:15.988259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:49:25.279880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.977250+0000","last_change":"2024-03-23T20:53:16.257895+0000","last_active":"2024-03-23T21:16:15.977250+0000","last_peered":"2024-03-23T21:16:15.977250+0000","last_clean":"2024-03-23T21:16:15.977250+0000","last_became_active":"2024-03-23T20:53:16.257435+0000","last_became_peered":"2024-03-23T20:53:16.257435+0000","last_unstale":"2024-03-23T21:16:15.977250+0000","last_undegraded":"2024-03-23T21:16:15.977250+0000","last_fullsized":"2024-03-23T21:16:15.977250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:20:33.429387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.473243+0000","last_change":"2024-03-23T20:53:16.259073+0000","last_active":"2024-03-23T21:16:15.473243+0000","last_peered":"2024-03-23T21:16:15.473243+0000","last_clean":"2024-03-23T21:16:15.473243+0000","last_became_active":"2024-03-23T20:53:16.258711+0000","last_became_peered":"2024-03-23T20:53:16.258711+0000","last_unstale":"2024-03-23T21:16:15.473243+0000","last_undegraded":"2024-03-23T21:16:15.473243+0000","last_fullsized":"2024-03-23T21:16:15.473243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:03:03.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":287,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.468653+0000","last_change":"2024-03-23T20:53:11.553674+0000","last_active":"2024-03-23T21:16:15.468653+0000","last_peered":"2024-03-23T21:16:15.468653+0000","last_clean":"2024-03-23T21:16:15.468653+0000","last_became_active":"2024-03-23T20:53:11.551476+0000","last_became_peered":"2024-03-23T20:53:11.551476+0000","last_unstale":"2024-03-23T21:16:15.468653+0000","last_undegraded":"2024-03-23T21:16:15.468653+0000","last_fullsized":"2024-03-23T21:16:15.468653+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_clean_scrub_stamp":"2024-03-23T20:53:10.128204+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:26.471264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738662,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40576,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331264,"statfs":{"total":96636764160,"available":96595214336,"internally_reserved":0,"allocated":1294336,"data_stored":647413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50600000000000001,"15min":0.503},"min":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.57699999999999996,"15min":0.54600000000000004},"min":{"1min":0.54100000000000004,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.72699999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51800000000000002}]},{"osd":1,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.373},"max":{"1min":0.75,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.69599999999999995,"5min":1.018,"15min":1.018},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738662,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1888256,"data_stored":1237781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.52200000000000002},"min":{"1min":0.434,"5min":0.40400000000000003,"15min":0.36199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61499999999999999,"15min":0.56499999999999995},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.39000000000000001},"max":{"1min":0.70299999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.498,"5min":0.44800000000000001,"15min":0.432},"max":{"1min":0.66500000000000004,"5min":0.76000000000000001,"15min":0.77200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.499,"5min":0.44700000000000001,"15min":0.42599999999999999},"max":{"1min":0.72199999999999998,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.61899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738660,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59196,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312644,"statfs":{"total":96636764160,"available":96576147456,"internally_reserved":0,"allocated":1884160,"data_stored":1237012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55300000000000005,"15min":0.52700000000000002},"min":{"1min":0.45600000000000002,"5min":0.42399999999999999,"15min":0.25700000000000001},"max":{"1min":0.747,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.876,"15min":0.876},"last":0.60199999999999998}]},{"osd":2,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.35299999999999998},"max":{"1min":0.67300000000000004,"5min":0.78400000000000003,"15min":1.0840000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.317},"max":{"1min":0.66600000000000004,"5min":1.456,"15min":1.456},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:17:16.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-23T21:17:16.311 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T21:17:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-23T21:17:16.329 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":872,"stamp":"2024-03-23T21:17:14.345428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":136828,"kb_used_data":4948,"kb_used_omap":4,"kb_used_meta":131835,"kb_avail":282978692,"statfs":{"total":289910292480,"available":289770180608,"internally_reserved":0,"allocated":5066752,"data_stored":3122206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":134999378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003380"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.478953+0000","last_change":"2024-03-23T20:53:16.260778+0000","last_active":"2024-03-23T21:16:17.478953+0000","last_peered":"2024-03-23T21:16:17.478953+0000","last_clean":"2024-03-23T21:16:17.478953+0000","last_became_active":"2024-03-23T20:53:16.260429+0000","last_became_peered":"2024-03-23T20:53:16.260429+0000","last_unstale":"2024-03-23T21:16:17.478953+0000","last_undegraded":"2024-03-23T21:16:17.478953+0000","last_fullsized":"2024-03-23T21:16:17.478953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:08:21.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.486080+0000","last_change":"2024-03-23T20:53:16.257714+0000","last_active":"2024-03-23T21:16:17.486080+0000","last_peered":"2024-03-23T21:16:17.486080+0000","last_clean":"2024-03-23T21:16:17.486080+0000","last_became_active":"2024-03-23T20:53:16.257287+0000","last_became_peered":"2024-03-23T20:53:16.257287+0000","last_unstale":"2024-03-23T21:16:17.486080+0000","last_undegraded":"2024-03-23T21:16:17.486080+0000","last_fullsized":"2024-03-23T21:16:17.486080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:25.116796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.483398+0000","last_change":"2024-03-23T20:53:16.260048+0000","last_active":"2024-03-23T21:16:17.483398+0000","last_peered":"2024-03-23T21:16:17.483398+0000","last_clean":"2024-03-23T21:16:17.483398+0000","last_became_active":"2024-03-23T20:53:16.259683+0000","last_became_peered":"2024-03-23T20:53:16.259683+0000","last_unstale":"2024-03-23T21:16:17.483398+0000","last_undegraded":"2024-03-23T21:16:17.483398+0000","last_fullsized":"2024-03-23T21:16:17.483398+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:39:53.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.465220+0000","last_change":"2024-03-23T20:53:16.260803+0000","last_active":"2024-03-23T21:16:15.465220+0000","last_peered":"2024-03-23T21:16:15.465220+0000","last_clean":"2024-03-23T21:16:15.465220+0000","last_became_active":"2024-03-23T20:53:16.260458+0000","last_became_peered":"2024-03-23T20:53:16.260458+0000","last_unstale":"2024-03-23T21:16:15.465220+0000","last_undegraded":"2024-03-23T21:16:15.465220+0000","last_fullsized":"2024-03-23T21:16:15.465220+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:25:51.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":156,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.656552+0000","last_change":"2024-03-23T20:53:16.258624+0000","last_active":"2024-03-23T21:16:15.656552+0000","last_peered":"2024-03-23T21:16:15.656552+0000","last_clean":"2024-03-23T21:16:15.656552+0000","last_became_active":"2024-03-23T20:53:16.257825+0000","last_became_peered":"2024-03-23T20:53:16.257825+0000","last_unstale":"2024-03-23T21:16:15.656552+0000","last_undegraded":"2024-03-23T21:16:15.656552+0000","last_fullsized":"2024-03-23T21:16:15.656552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:55.573156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.988259+0000","last_change":"2024-03-23T20:53:16.257583+0000","last_active":"2024-03-23T21:16:15.988259+0000","last_peered":"2024-03-23T21:16:15.988259+0000","last_clean":"2024-03-23T21:16:15.988259+0000","last_became_active":"2024-03-23T20:53:16.257206+0000","last_became_peered":"2024-03-23T20:53:16.257206+0000","last_unstale":"2024-03-23T21:16:15.988259+0000","last_undegraded":"2024-03-23T21:16:15.988259+0000","last_fullsized":"2024-03-23T21:16:15.988259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:49:25.279880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.977250+0000","last_change":"2024-03-23T20:53:16.257895+0000","last_active":"2024-03-23T21:16:15.977250+0000","last_peered":"2024-03-23T21:16:15.977250+0000","last_clean":"2024-03-23T21:16:15.977250+0000","last_became_active":"2024-03-23T20:53:16.257435+0000","last_became_peered":"2024-03-23T20:53:16.257435+0000","last_unstale":"2024-03-23T21:16:15.977250+0000","last_undegraded":"2024-03-23T21:16:15.977250+0000","last_fullsized":"2024-03-23T21:16:15.977250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:20:33.429387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.473243+0000","last_change":"2024-03-23T20:53:16.259073+0000","last_active":"2024-03-23T21:16:15.473243+0000","last_peered":"2024-03-23T21:16:15.473243+0000","last_clean":"2024-03-23T21:16:15.473243+0000","last_became_active":"2024-03-23T20:53:16.258711+0000","last_became_peered":"2024-03-23T20:53:16.258711+0000","last_unstale":"2024-03-23T21:16:15.473243+0000","last_undegraded":"2024-03-23T21:16:15.473243+0000","last_fullsized":"2024-03-23T21:16:15.473243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:03:03.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":287,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.468653+0000","last_change":"2024-03-23T20:53:11.553674+0000","last_active":"2024-03-23T21:16:15.468653+0000","last_peered":"2024-03-23T21:16:15.468653+0000","last_clean":"2024-03-23T21:16:15.468653+0000","last_became_active":"2024-03-23T20:53:11.551476+0000","last_became_peered":"2024-03-23T20:53:11.551476+0000","last_unstale":"2024-03-23T21:16:15.468653+0000","last_undegraded":"2024-03-23T21:16:15.468653+0000","last_fullsized":"2024-03-23T21:16:15.468653+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_clean_scrub_stamp":"2024-03-23T20:53:10.128204+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:26.471264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738662,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40576,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331264,"statfs":{"total":96636764160,"available":96595214336,"internally_reserved":0,"allocated":1294336,"data_stored":647413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50600000000000001,"15min":0.503},"min":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.57699999999999996,"15min":0.54600000000000004},"min":{"1min":0.54100000000000004,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.72699999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51800000000000002}]},{"osd":1,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.373},"max":{"1min":0.75,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.69599999999999995,"5min":1.018,"15min":1.018},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738662,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1888256,"data_stored":1237781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.52200000000000002},"min":{"1min":0.434,"5min":0.40400000000000003,"15min":0.36199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61499999999999999,"15min":0.56499999999999995},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.39000000000000001},"max":{"1min":0.70299999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.498,"5min":0.44800000000000001,"15min":0.432},"max":{"1min":0.66500000000000004,"5min":0.76000000000000001,"15min":0.77200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.499,"5min":0.44700000000000001,"15min":0.42599999999999999},"max":{"1min":0.72199999999999998,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.61899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738660,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59196,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312644,"statfs":{"total":96636764160,"available":96576147456,"internally_reserved":0,"allocated":1884160,"data_stored":1237012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55300000000000005,"15min":0.52700000000000002},"min":{"1min":0.45600000000000002,"5min":0.42399999999999999,"15min":0.25700000000000001},"max":{"1min":0.747,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.876,"15min":0.876},"last":0.60199999999999998}]},{"osd":2,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.35299999999999998},"max":{"1min":0.67300000000000004,"5min":0.78400000000000003,"15min":1.0840000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.317},"max":{"1min":0.66600000000000004,"5min":1.456,"15min":1.456},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:17:16.330 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-03-23T21:17:16.330 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-23T21:17:16.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-03-23T21:17:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T21:17:16.612 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-23T21:17:16.629 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":873,"stamp":"2024-03-23T21:17:16.346015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":136828,"kb_used_data":4948,"kb_used_omap":4,"kb_used_meta":131835,"kb_avail":282978692,"statfs":{"total":289910292480,"available":289770180608,"internally_reserved":0,"allocated":5066752,"data_stored":3122206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":134999378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003494"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.478953+0000","last_change":"2024-03-23T20:53:16.260778+0000","last_active":"2024-03-23T21:16:17.478953+0000","last_peered":"2024-03-23T21:16:17.478953+0000","last_clean":"2024-03-23T21:16:17.478953+0000","last_became_active":"2024-03-23T20:53:16.260429+0000","last_became_peered":"2024-03-23T20:53:16.260429+0000","last_unstale":"2024-03-23T21:16:17.478953+0000","last_undegraded":"2024-03-23T21:16:17.478953+0000","last_fullsized":"2024-03-23T21:16:17.478953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:08:21.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.486080+0000","last_change":"2024-03-23T20:53:16.257714+0000","last_active":"2024-03-23T21:16:17.486080+0000","last_peered":"2024-03-23T21:16:17.486080+0000","last_clean":"2024-03-23T21:16:17.486080+0000","last_became_active":"2024-03-23T20:53:16.257287+0000","last_became_peered":"2024-03-23T20:53:16.257287+0000","last_unstale":"2024-03-23T21:16:17.486080+0000","last_undegraded":"2024-03-23T21:16:17.486080+0000","last_fullsized":"2024-03-23T21:16:17.486080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:25.116796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.483398+0000","last_change":"2024-03-23T20:53:16.260048+0000","last_active":"2024-03-23T21:16:17.483398+0000","last_peered":"2024-03-23T21:16:17.483398+0000","last_clean":"2024-03-23T21:16:17.483398+0000","last_became_active":"2024-03-23T20:53:16.259683+0000","last_became_peered":"2024-03-23T20:53:16.259683+0000","last_unstale":"2024-03-23T21:16:17.483398+0000","last_undegraded":"2024-03-23T21:16:17.483398+0000","last_fullsized":"2024-03-23T21:16:17.483398+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:39:53.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.465220+0000","last_change":"2024-03-23T20:53:16.260803+0000","last_active":"2024-03-23T21:16:15.465220+0000","last_peered":"2024-03-23T21:16:15.465220+0000","last_clean":"2024-03-23T21:16:15.465220+0000","last_became_active":"2024-03-23T20:53:16.260458+0000","last_became_peered":"2024-03-23T20:53:16.260458+0000","last_unstale":"2024-03-23T21:16:15.465220+0000","last_undegraded":"2024-03-23T21:16:15.465220+0000","last_fullsized":"2024-03-23T21:16:15.465220+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:25:51.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":156,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.656552+0000","last_change":"2024-03-23T20:53:16.258624+0000","last_active":"2024-03-23T21:16:15.656552+0000","last_peered":"2024-03-23T21:16:15.656552+0000","last_clean":"2024-03-23T21:16:15.656552+0000","last_became_active":"2024-03-23T20:53:16.257825+0000","last_became_peered":"2024-03-23T20:53:16.257825+0000","last_unstale":"2024-03-23T21:16:15.656552+0000","last_undegraded":"2024-03-23T21:16:15.656552+0000","last_fullsized":"2024-03-23T21:16:15.656552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:55.573156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.988259+0000","last_change":"2024-03-23T20:53:16.257583+0000","last_active":"2024-03-23T21:16:15.988259+0000","last_peered":"2024-03-23T21:16:15.988259+0000","last_clean":"2024-03-23T21:16:15.988259+0000","last_became_active":"2024-03-23T20:53:16.257206+0000","last_became_peered":"2024-03-23T20:53:16.257206+0000","last_unstale":"2024-03-23T21:16:15.988259+0000","last_undegraded":"2024-03-23T21:16:15.988259+0000","last_fullsized":"2024-03-23T21:16:15.988259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:49:25.279880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.977250+0000","last_change":"2024-03-23T20:53:16.257895+0000","last_active":"2024-03-23T21:16:15.977250+0000","last_peered":"2024-03-23T21:16:15.977250+0000","last_clean":"2024-03-23T21:16:15.977250+0000","last_became_active":"2024-03-23T20:53:16.257435+0000","last_became_peered":"2024-03-23T20:53:16.257435+0000","last_unstale":"2024-03-23T21:16:15.977250+0000","last_undegraded":"2024-03-23T21:16:15.977250+0000","last_fullsized":"2024-03-23T21:16:15.977250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:20:33.429387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.473243+0000","last_change":"2024-03-23T20:53:16.259073+0000","last_active":"2024-03-23T21:16:15.473243+0000","last_peered":"2024-03-23T21:16:15.473243+0000","last_clean":"2024-03-23T21:16:15.473243+0000","last_became_active":"2024-03-23T20:53:16.258711+0000","last_became_peered":"2024-03-23T20:53:16.258711+0000","last_unstale":"2024-03-23T21:16:15.473243+0000","last_undegraded":"2024-03-23T21:16:15.473243+0000","last_fullsized":"2024-03-23T21:16:15.473243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:03:03.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":287,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.468653+0000","last_change":"2024-03-23T20:53:11.553674+0000","last_active":"2024-03-23T21:16:15.468653+0000","last_peered":"2024-03-23T21:16:15.468653+0000","last_clean":"2024-03-23T21:16:15.468653+0000","last_became_active":"2024-03-23T20:53:11.551476+0000","last_became_peered":"2024-03-23T20:53:11.551476+0000","last_unstale":"2024-03-23T21:16:15.468653+0000","last_undegraded":"2024-03-23T21:16:15.468653+0000","last_fullsized":"2024-03-23T21:16:15.468653+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_clean_scrub_stamp":"2024-03-23T20:53:10.128204+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:26.471264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738662,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40576,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331264,"statfs":{"total":96636764160,"available":96595214336,"internally_reserved":0,"allocated":1294336,"data_stored":647413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50600000000000001,"15min":0.503},"min":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.57699999999999996,"15min":0.54600000000000004},"min":{"1min":0.54100000000000004,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.72699999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51800000000000002}]},{"osd":1,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.373},"max":{"1min":0.75,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.69599999999999995,"5min":1.018,"15min":1.018},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738662,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1888256,"data_stored":1237781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.52200000000000002},"min":{"1min":0.434,"5min":0.40400000000000003,"15min":0.36199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61499999999999999,"15min":0.56499999999999995},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.39000000000000001},"max":{"1min":0.70299999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.498,"5min":0.44800000000000001,"15min":0.432},"max":{"1min":0.66500000000000004,"5min":0.76000000000000001,"15min":0.77200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.499,"5min":0.44700000000000001,"15min":0.42599999999999999},"max":{"1min":0.72199999999999998,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.61899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738661,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59196,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312644,"statfs":{"total":96636764160,"available":96576147456,"internally_reserved":0,"allocated":1884160,"data_stored":1237012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55300000000000005,"15min":0.52700000000000002},"min":{"1min":0.45600000000000002,"5min":0.42399999999999999,"15min":0.25700000000000001},"max":{"1min":0.747,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.876,"15min":0.876},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.35299999999999998},"max":{"1min":0.67300000000000004,"5min":0.78400000000000003,"15min":1.0840000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.317},"max":{"1min":0.66600000000000004,"5min":1.456,"15min":1.456},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:17:16.630 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-03-23T21:17:16.913 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T21:17:16.913 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":151,"fsid":"f60222f0-44b2-480a-bdc8-0eb22ebe7b33","created":"2024-03-23T20:52:59.149423+0000","modified":"2024-03-23T21:16:17.464190+0000","last_up_change":"2024-03-23T20:53:09.102717+0000","last_in_change":"2024-03-23T20:53:00.037665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T20:53:09.899996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-23T20:53:14.441478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0363173-1820-4027-8318-a96cb3f0e5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23953},{"type":"v1","addr":"172.21.15.117:6817","nonce":23953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23953},{"type":"v1","addr":"172.21.15.117:6819","nonce":23953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23953},{"type":"v1","addr":"172.21.15.117:6823","nonce":23953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23953},{"type":"v1","addr":"172.21.15.117:6821","nonce":23953}]},"public_addr":"172.21.15.117:6817/23953","cluster_addr":"172.21.15.117:6819/23953","heartbeat_back_addr":"172.21.15.117:6823/23953","heartbeat_front_addr":"172.21.15.117:6821/23953","state":["exists","up"]},{"osd":1,"uuid":"c6028d94-1be7-43bb-8c5c-999d150aa717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23952},{"type":"v1","addr":"172.21.15.117:6809","nonce":23952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23952},{"type":"v1","addr":"172.21.15.117:6811","nonce":23952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23952},{"type":"v1","addr":"172.21.15.117:6815","nonce":23952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23952},{"type":"v1","addr":"172.21.15.117:6813","nonce":23952}]},"public_addr":"172.21.15.117:6809/23952","cluster_addr":"172.21.15.117:6811/23952","heartbeat_back_addr":"172.21.15.117:6815/23952","heartbeat_front_addr":"172.21.15.117:6813/23952","state":["exists","up"]},{"osd":2,"uuid":"8595a61f-9a3a-4503-9214-1f7dd7d385a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":23954},{"type":"v1","addr":"172.21.15.117:6801","nonce":23954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23954},{"type":"v1","addr":"172.21.15.117:6803","nonce":23954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23954},{"type":"v1","addr":"172.21.15.117:6807","nonce":23954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23954},{"type":"v1","addr":"172.21.15.117:6805","nonce":23954}]},"public_addr":"172.21.15.117:6801/23954","cluster_addr":"172.21.15.117:6803/23954","heartbeat_back_addr":"172.21.15.117:6807/23954","heartbeat_front_addr":"172.21.15.117:6805/23954","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.628809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.575003+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T20:53:05.402693+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":1},{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T21:17:17.931 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-03-23T21:17:17.931 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-23T21:17:18.085 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-23T21:17:18.086 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T21:17:18.086 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-23T21:17:18.102 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-03-23T21:17:18.373 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 0 to deep-scrub 2024-03-23T21:17:18.390 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-03-23T21:17:18.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-23T21:17:18.528 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-23T21:17:18.528 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T21:17:18.528 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-23T21:17:18.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-03-23T21:17:18.816 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 1 to deep-scrub 2024-03-23T21:17:18.833 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-03-23T21:17:18.833 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-23T21:17:18.967 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-23T21:17:18.967 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T21:17:18.967 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-23T21:17:18.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-03-23T21:17:19.261 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 2 to deep-scrub 2024-03-23T21:17:19.280 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-23T21:17:19.542 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T21:17:19.542 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-23T21:17:19.557 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":874,"stamp":"2024-03-23T21:17:18.346348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":136828,"kb_used_data":4948,"kb_used_omap":4,"kb_used_meta":131835,"kb_avail":282978692,"statfs":{"total":289910292480,"available":289770180608,"internally_reserved":0,"allocated":5066752,"data_stored":3122206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":134999378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003312"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.478953+0000","last_change":"2024-03-23T20:53:16.260778+0000","last_active":"2024-03-23T21:16:17.478953+0000","last_peered":"2024-03-23T21:16:17.478953+0000","last_clean":"2024-03-23T21:16:17.478953+0000","last_became_active":"2024-03-23T20:53:16.260429+0000","last_became_peered":"2024-03-23T20:53:16.260429+0000","last_unstale":"2024-03-23T21:16:17.478953+0000","last_undegraded":"2024-03-23T21:16:17.478953+0000","last_fullsized":"2024-03-23T21:16:17.478953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:08:21.075585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.486080+0000","last_change":"2024-03-23T20:53:16.257714+0000","last_active":"2024-03-23T21:16:17.486080+0000","last_peered":"2024-03-23T21:16:17.486080+0000","last_clean":"2024-03-23T21:16:17.486080+0000","last_became_active":"2024-03-23T20:53:16.257287+0000","last_became_peered":"2024-03-23T20:53:16.257287+0000","last_unstale":"2024-03-23T21:16:17.486080+0000","last_undegraded":"2024-03-23T21:16:17.486080+0000","last_fullsized":"2024-03-23T21:16:17.486080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:41:25.116796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:16:17.483398+0000","last_change":"2024-03-23T20:53:16.260048+0000","last_active":"2024-03-23T21:16:17.483398+0000","last_peered":"2024-03-23T21:16:17.483398+0000","last_clean":"2024-03-23T21:16:17.483398+0000","last_became_active":"2024-03-23T20:53:16.259683+0000","last_became_peered":"2024-03-23T20:53:16.259683+0000","last_unstale":"2024-03-23T21:16:17.483398+0000","last_undegraded":"2024-03-23T21:16:17.483398+0000","last_fullsized":"2024-03-23T21:16:17.483398+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:39:53.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.465220+0000","last_change":"2024-03-23T20:53:16.260803+0000","last_active":"2024-03-23T21:16:15.465220+0000","last_peered":"2024-03-23T21:16:15.465220+0000","last_clean":"2024-03-23T21:16:15.465220+0000","last_became_active":"2024-03-23T20:53:16.260458+0000","last_became_peered":"2024-03-23T20:53:16.260458+0000","last_unstale":"2024-03-23T21:16:15.465220+0000","last_undegraded":"2024-03-23T21:16:15.465220+0000","last_fullsized":"2024-03-23T21:16:15.465220+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:25:51.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":156,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.656552+0000","last_change":"2024-03-23T20:53:16.258624+0000","last_active":"2024-03-23T21:16:15.656552+0000","last_peered":"2024-03-23T21:16:15.656552+0000","last_clean":"2024-03-23T21:16:15.656552+0000","last_became_active":"2024-03-23T20:53:16.257825+0000","last_became_peered":"2024-03-23T20:53:16.257825+0000","last_unstale":"2024-03-23T21:16:15.656552+0000","last_undegraded":"2024-03-23T21:16:15.656552+0000","last_fullsized":"2024-03-23T21:16:15.656552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:55.573156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.988259+0000","last_change":"2024-03-23T20:53:16.257583+0000","last_active":"2024-03-23T21:16:15.988259+0000","last_peered":"2024-03-23T21:16:15.988259+0000","last_clean":"2024-03-23T21:16:15.988259+0000","last_became_active":"2024-03-23T20:53:16.257206+0000","last_became_peered":"2024-03-23T20:53:16.257206+0000","last_unstale":"2024-03-23T21:16:15.988259+0000","last_undegraded":"2024-03-23T21:16:15.988259+0000","last_fullsized":"2024-03-23T21:16:15.988259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:49:25.279880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.977250+0000","last_change":"2024-03-23T20:53:16.257895+0000","last_active":"2024-03-23T21:16:15.977250+0000","last_peered":"2024-03-23T21:16:15.977250+0000","last_clean":"2024-03-23T21:16:15.977250+0000","last_became_active":"2024-03-23T20:53:16.257435+0000","last_became_peered":"2024-03-23T20:53:16.257435+0000","last_unstale":"2024-03-23T21:16:15.977250+0000","last_undegraded":"2024-03-23T21:16:15.977250+0000","last_fullsized":"2024-03-23T21:16:15.977250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:20:33.429387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":157,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.473243+0000","last_change":"2024-03-23T20:53:16.259073+0000","last_active":"2024-03-23T21:16:15.473243+0000","last_peered":"2024-03-23T21:16:15.473243+0000","last_clean":"2024-03-23T21:16:15.473243+0000","last_became_active":"2024-03-23T20:53:16.258711+0000","last_became_peered":"2024-03-23T20:53:16.258711+0000","last_unstale":"2024-03-23T21:16:15.473243+0000","last_undegraded":"2024-03-23T21:16:15.473243+0000","last_fullsized":"2024-03-23T21:16:15.473243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:15.225789+0000","last_clean_scrub_stamp":"2024-03-23T20:53:15.225789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:03:03.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":287,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-03-23T21:16:15.468653+0000","last_change":"2024-03-23T20:53:11.553674+0000","last_active":"2024-03-23T21:16:15.468653+0000","last_peered":"2024-03-23T21:16:15.468653+0000","last_clean":"2024-03-23T21:16:15.468653+0000","last_became_active":"2024-03-23T20:53:11.551476+0000","last_became_peered":"2024-03-23T20:53:11.551476+0000","last_unstale":"2024-03-23T21:16:15.468653+0000","last_undegraded":"2024-03-23T21:16:15.468653+0000","last_fullsized":"2024-03-23T21:16:15.468653+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T20:53:10.128204+0000","last_clean_scrub_stamp":"2024-03-23T20:53:10.128204+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:26.471264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738662,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40576,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331264,"statfs":{"total":96636764160,"available":96595214336,"internally_reserved":0,"allocated":1294336,"data_stored":647413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50600000000000001,"15min":0.503},"min":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.57699999999999996,"15min":0.54600000000000004},"min":{"1min":0.54100000000000004,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.72699999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51800000000000002}]},{"osd":1,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.373},"max":{"1min":0.75,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.69599999999999995,"5min":1.018,"15min":1.018},"last":0.47699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738662,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1888256,"data_stored":1237781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.52200000000000002},"min":{"1min":0.434,"5min":0.40400000000000003,"15min":0.36199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61499999999999999,"15min":0.56499999999999995},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.39000000000000001},"max":{"1min":0.70299999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.498,"5min":0.44800000000000001,"15min":0.432},"max":{"1min":0.66500000000000004,"5min":0.76000000000000001,"15min":0.77200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.499,"5min":0.44700000000000001,"15min":0.42599999999999999},"max":{"1min":0.72199999999999998,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.61899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738661,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59196,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312644,"statfs":{"total":96636764160,"available":96576147456,"internally_reserved":0,"allocated":1884160,"data_stored":1237012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55300000000000005,"15min":0.52700000000000002},"min":{"1min":0.45600000000000002,"5min":0.42399999999999999,"15min":0.25700000000000001},"max":{"1min":0.747,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.876,"15min":0.876},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.35299999999999998},"max":{"1min":0.67300000000000004,"5min":0.78400000000000003,"15min":1.0840000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.317},"max":{"1min":0.66600000000000004,"5min":1.456,"15min":1.456},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:17:19.558 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-23T20:53:15.225789+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=15, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-23T20:53:15.225789+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=15, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-23T20:53:15.225789+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=15, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-23T20:53:15.225789+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=15, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-23T20:53:15.225789+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=15, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-23T20:53:15.225789+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=15, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-23T20:53:15.225789+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=15, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-23T20:53:15.225789+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=15, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-23T20:53:10.128204+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=20, tm_min=53, tm_sec=10, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=21, tm_min=17, tm_sec=16, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T21:17:19.559 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-23T21:17:39.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-03-23T21:17:39.859 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-23T21:17:39.859 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-23T21:17:39.875 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-03-23T21:17:38.352039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":136812,"kb_used_data":4948,"kb_used_omap":4,"kb_used_meta":131835,"kb_avail":282978708,"statfs":{"total":289910292480,"available":289770196992,"internally_reserved":0,"allocated":5066752,"data_stored":3122206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":134999378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:22.497864+0000","last_change":"2024-03-23T21:17:22.498131+0000","last_active":"2024-03-23T21:17:22.497864+0000","last_peered":"2024-03-23T21:17:22.497864+0000","last_clean":"2024-03-23T21:17:22.497864+0000","last_became_active":"2024-03-23T20:53:16.260429+0000","last_became_peered":"2024-03-23T20:53:16.260429+0000","last_unstale":"2024-03-23T21:17:22.497864+0000","last_undegraded":"2024-03-23T21:17:22.497864+0000","last_fullsized":"2024-03-23T21:17:22.497864+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:17:22.497819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:17:22.497819+0000","last_clean_scrub_stamp":"2024-03-23T21:17:22.497819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:54:20.778375+0000","scrub_duration":0.0064359710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:23.462390+0000","last_change":"2024-03-23T21:17:23.462613+0000","last_active":"2024-03-23T21:17:23.462390+0000","last_peered":"2024-03-23T21:17:23.462390+0000","last_clean":"2024-03-23T21:17:23.462390+0000","last_became_active":"2024-03-23T20:53:16.257287+0000","last_became_peered":"2024-03-23T20:53:16.257287+0000","last_unstale":"2024-03-23T21:17:23.462390+0000","last_undegraded":"2024-03-23T21:17:23.462390+0000","last_fullsized":"2024-03-23T21:17:23.462390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:17:23.462338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:17:23.462338+0000","last_clean_scrub_stamp":"2024-03-23T21:17:23.462338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:35:48.258710+0000","scrub_duration":0.00383673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:26.526717+0000","last_change":"2024-03-23T21:17:26.526958+0000","last_active":"2024-03-23T21:17:26.526717+0000","last_peered":"2024-03-23T21:17:26.526717+0000","last_clean":"2024-03-23T21:17:26.526717+0000","last_became_active":"2024-03-23T20:53:16.259683+0000","last_became_peered":"2024-03-23T20:53:16.259683+0000","last_unstale":"2024-03-23T21:17:26.526717+0000","last_undegraded":"2024-03-23T21:17:26.526717+0000","last_fullsized":"2024-03-23T21:17:26.526717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:17:26.526665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:17:26.526665+0000","last_clean_scrub_stamp":"2024-03-23T21:17:26.526665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:41.257597+0000","scrub_duration":0.0053939160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:28.583382+0000","last_change":"2024-03-23T21:17:28.583604+0000","last_active":"2024-03-23T21:17:28.583382+0000","last_peered":"2024-03-23T21:17:28.583382+0000","last_clean":"2024-03-23T21:17:28.583382+0000","last_became_active":"2024-03-23T20:53:16.260458+0000","last_became_peered":"2024-03-23T20:53:16.260458+0000","last_unstale":"2024-03-23T21:17:28.583382+0000","last_undegraded":"2024-03-23T21:17:28.583382+0000","last_fullsized":"2024-03-23T21:17:28.583382+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:17:28.583325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:17:28.583325+0000","last_clean_scrub_stamp":"2024-03-23T21:17:28.583325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:18.982774+0000","scrub_duration":0.0028151109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:18.646491+0000","last_change":"2024-03-23T21:17:18.646574+0000","last_active":"2024-03-23T21:17:18.646491+0000","last_peered":"2024-03-23T21:17:18.646491+0000","last_clean":"2024-03-23T21:17:18.646491+0000","last_became_active":"2024-03-23T20:53:16.257825+0000","last_became_peered":"2024-03-23T20:53:16.257825+0000","last_unstale":"2024-03-23T21:17:18.646491+0000","last_undegraded":"2024-03-23T21:17:18.646491+0000","last_fullsized":"2024-03-23T21:17:18.646491+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:17:18.646472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:17:18.646472+0000","last_clean_scrub_stamp":"2024-03-23T21:17:18.646472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:51.428817+0000","scrub_duration":0.0015457750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:23.174208+0000","last_change":"2024-03-23T21:17:23.174425+0000","last_active":"2024-03-23T21:17:23.174208+0000","last_peered":"2024-03-23T21:17:23.174208+0000","last_clean":"2024-03-23T21:17:23.174208+0000","last_became_active":"2024-03-23T20:53:16.257206+0000","last_became_peered":"2024-03-23T20:53:16.257206+0000","last_unstale":"2024-03-23T21:17:23.174208+0000","last_undegraded":"2024-03-23T21:17:23.174208+0000","last_fullsized":"2024-03-23T21:17:23.174208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:17:23.174163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:17:23.174163+0000","last_clean_scrub_stamp":"2024-03-23T21:17:23.174163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:52.959551+0000","scrub_duration":0.0049568909999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":170,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:21.142853+0000","last_change":"2024-03-23T21:17:21.142853+0000","last_active":"2024-03-23T21:17:21.142853+0000","last_peered":"2024-03-23T21:17:21.142853+0000","last_clean":"2024-03-23T21:17:21.142853+0000","last_became_active":"2024-03-23T20:53:16.257435+0000","last_became_peered":"2024-03-23T20:53:16.257435+0000","last_unstale":"2024-03-23T21:17:21.142853+0000","last_undegraded":"2024-03-23T21:17:21.142853+0000","last_fullsized":"2024-03-23T21:17:21.142853+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:17:21.142595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:17:21.142595+0000","last_clean_scrub_stamp":"2024-03-23T21:17:21.142595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:01:25.624487+0000","scrub_duration":0.00382578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:25.488633+0000","last_change":"2024-03-23T21:17:25.488852+0000","last_active":"2024-03-23T21:17:25.488633+0000","last_peered":"2024-03-23T21:17:25.488633+0000","last_clean":"2024-03-23T21:17:25.488633+0000","last_became_active":"2024-03-23T20:53:16.258711+0000","last_became_peered":"2024-03-23T20:53:16.258711+0000","last_unstale":"2024-03-23T21:17:25.488633+0000","last_undegraded":"2024-03-23T21:17:25.488633+0000","last_fullsized":"2024-03-23T21:17:25.488633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T21:17:25.488586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T21:17:25.488586+0000","last_clean_scrub_stamp":"2024-03-23T21:17:25.488586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:38:53.598496+0000","scrub_duration":0.0058469209999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"151'99","reported_seq":302,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-23T21:17:20.490397+0000","last_change":"2024-03-23T21:17:20.490710+0000","last_active":"2024-03-23T21:17:20.490397+0000","last_peered":"2024-03-23T21:17:20.490397+0000","last_clean":"2024-03-23T21:17:20.490397+0000","last_became_active":"2024-03-23T20:53:11.551476+0000","last_became_peered":"2024-03-23T20:53:11.551476+0000","last_unstale":"2024-03-23T21:17:20.490397+0000","last_undegraded":"2024-03-23T21:17:20.490397+0000","last_fullsized":"2024-03-23T21:17:20.490397+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'99","last_scrub_stamp":"2024-03-23T21:17:20.490337+0000","last_deep_scrub":"151'99","last_deep_scrub_stamp":"2024-03-23T21:17:20.490337+0000","last_clean_scrub_stamp":"2024-03-23T21:17:20.490337+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:38.953388+0000","scrub_duration":0.0097960900000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40568,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331272,"statfs":{"total":96636764160,"available":96595222528,"internally_reserved":0,"allocated":1294336,"data_stored":647413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50600000000000001,"15min":0.503},"min":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.373},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.57699999999999996,"15min":0.54600000000000004},"min":{"1min":0.54100000000000004,"5min":0.42899999999999999,"15min":0.41099999999999998},"max":{"1min":0.72699999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.66500000000000004}]},{"osd":1,"last update":"Sat Mar 23 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.373},"max":{"1min":0.75,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.626},"min":{"1min":0.48799999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.69599999999999995,"5min":1.018,"15min":1.018},"last":0.45600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738666,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37052,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334788,"statfs":{"total":96636764160,"available":96598822912,"internally_reserved":0,"allocated":1888256,"data_stored":1237781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53600000000000003,"15min":0.52200000000000002},"min":{"1min":0.434,"5min":0.40400000000000003,"15min":0.36199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61499999999999999,"15min":0.56499999999999995},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.39000000000000001},"max":{"1min":0.70299999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.67800000000000005}]},{"osd":2,"last update":"Sat Mar 23 21:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.498,"5min":0.44800000000000001,"15min":0.432},"max":{"1min":0.66500000000000004,"5min":0.76000000000000001,"15min":0.77200000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.59299999999999997},"min":{"1min":0.499,"5min":0.44700000000000001,"15min":0.42599999999999999},"max":{"1min":0.72199999999999998,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59192,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312648,"statfs":{"total":96636764160,"available":96576151552,"internally_reserved":0,"allocated":1884160,"data_stored":1237012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55300000000000005,"15min":0.52700000000000002},"min":{"1min":0.45600000000000002,"5min":0.42399999999999999,"15min":0.25700000000000001},"max":{"1min":0.747,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.28100000000000003},"max":{"1min":0.626,"5min":0.876,"15min":0.876},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Mar 23 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.55600000000000005},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.35299999999999998},"max":{"1min":0.67300000000000004,"5min":0.78400000000000003,"15min":1.0840000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.59299999999999997},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.317},"max":{"1min":0.66600000000000004,"5min":1.456,"15min":1.456},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T21:17:39.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-03-23T21:17:40.193 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-23T21:17:40.194 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-23T21:17:40.194 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-03-23T21:17:40.194 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:40.277 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-03-23T21:17:40.277 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-03-23T21:17:40.277 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:40.367 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-03-23T21:17:40.368 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-03-23T21:17:40.368 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:40.437 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-03-23T21:17:40.437 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-23T21:17:40.438 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-03-23T21:17:40.438 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:40.471 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-03-23T21:17:40.471 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-23T21:17:40.471 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-03-23T21:17:40.472 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T21:17:40.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-03-23T21:17:40.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.mon.a is failed for ~0s 2024-03-23T21:17:40.527 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-03-23T21:17:40.527 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-23T21:17:40.527 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-03-23T21:17:40.582 INFO:teuthology.orchestra.run.smithi117.stdout:2024-03-23T20:55:05.911428+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: Degraded data redundancy: 1/22 objects degraded (4.545%), 1 pg degraded (PG_DEGRADED) 2024-03-23T21:17:40.583 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-23T21:17:40.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-03-23T21:17:40.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-03-23T21:17:40.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[WRN\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-03-23T21:17:40.748 INFO:teuthology.orchestra.run.smithi117.stdout:2024-03-23T20:55:05.911428+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: Degraded data redundancy: 1/22 objects degraded (4.545%), 1 pg degraded (PG_DEGRADED) 2024-03-23T21:17:40.749 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T21:17:40.749 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-03-23T21:17:41.050 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T21:17:41.050 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-03-23T21:17:41.230 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T21:17:41.231 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-03-23T21:17:41.387 INFO:tasks.ceph:Archiving mon data... 2024-03-23T21:17:41.387 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/data/cluster1.mon.a.tgz 2024-03-23T21:17:41.387 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-23T21:17:41.392 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.5uR5IGOCfw 2024-03-23T21:17:41.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.5uR5IGOCfw 2024-03-23T21:17:41.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.5uR5IGOCfw 2024-03-23T21:17:41.594 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.5uR5IGOCfw is 381KB 2024-03-23T21:17:41.656 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.5uR5IGOCfw 2024-03-23T21:17:41.662 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-23T21:17:41.662 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T21:17:41.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T21:17:41.722 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-03-23T21:17:41.722 DEBUG:teuthology.orchestra.run.smithi117:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-23T21:17:41.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-23T21:17:41.788 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-23T21:17:41.795 INFO:teuthology.orchestra.run.smithi204.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-23T21:17:41.798 INFO:tasks.ceph:Archiving crash dumps... 2024-03-23T21:17:41.800 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/remote/smithi117/crash 2024-03-23T21:17:41.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T21:17:41.848 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/remote/smithi204/crash 2024-03-23T21:17:41.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T21:17:41.859 INFO:tasks.ceph:Compressing logs... 2024-03-23T21:17:41.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T21:17:41.895 DEBUG:teuthology.orchestra.run.smithi204:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T21:17:45.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-03-23T21:17:45.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.mon.a has been restored 2024-03-23T21:17:56.865 INFO:tasks.ceph:Archiving logs... 2024-03-23T21:17:56.865 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/remote/smithi117/log 2024-03-23T21:17:56.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T21:17:58.223 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/remote/smithi204/log 2024-03-23T21:17:58.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T21:18:00.578 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-23T21:18:00.593 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-23T21:18:00.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T21:18:00.596 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T21:18:00.664 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-23T21:18:00.673 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-23T21:18:00.673 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T21:18:00.676 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T21:18:00.685 INFO:teuthology.orchestra.run.smithi117.stdout: remote refid st t when poll reach delay offset jitter 2024-03-23T21:18:00.685 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================== 2024-03-23T21:18:00.686 INFO:teuthology.orchestra.run.smithi117.stdout:*hv01.front.sepi 44.4.53.2 2 u 42 64 377 0.107 -6.897 0.817 2024-03-23T21:18:00.686 INFO:teuthology.orchestra.run.smithi117.stdout:+hv02.front.sepi 63.231.80.2 3 u 43 64 377 0.092 -5.470 0.864 2024-03-23T21:18:00.686 INFO:teuthology.orchestra.run.smithi117.stdout:+hv03.front.sepi 74.6.168.72 3 u 45 64 377 0.083 -6.804 0.733 2024-03-23T21:18:00.686 INFO:teuthology.orchestra.run.smithi117.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-23T21:18:00.689 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2024-03-23T21:18:00.690 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2024-03-23T21:18:00.690 INFO:teuthology.orchestra.run.smithi204.stdout:*hv01.front.sepi 44.4.53.2 2 u 59 64 377 0.078 -2.775 0.294 2024-03-23T21:18:00.690 INFO:teuthology.orchestra.run.smithi204.stdout:+hv02.front.sepi 63.231.80.2 3 u 43 64 377 0.082 -1.947 0.254 2024-03-23T21:18:00.690 INFO:teuthology.orchestra.run.smithi204.stdout:+hv03.front.sepi 74.6.168.72 3 u 42 64 377 0.079 -2.969 0.257 2024-03-23T21:18:00.690 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-23T21:18:00.691 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-23T21:18:00.700 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-23T21:18:00.700 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-23T21:18:00.709 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-23T21:18:00.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-23T21:18:00.728 INFO:teuthology.task.internal:Duration was 2056.446011 seconds 2024-03-23T21:18:00.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-23T21:18:00.737 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-23T21:18:00.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T21:18:00.739 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T21:18:00.778 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-23T21:18:00.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-03-23T21:18:00.778 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T21:18:00.822 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-03-23T21:18:00.822 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T21:18:00.836 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-23T21:18:00.837 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T21:18:00.867 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T21:18:00.905 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-23T21:18:00.905 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T21:18:00.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T21:18:01.007 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-23T21:18:01.017 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-23T21:18:01.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T21:18:01.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T21:18:01.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-23T21:18:01.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T21:18:01.070 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T21:18:01.079 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-03-23T21:18:01.093 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-03-23T21:18:01.171 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T21:18:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T21:18:01.218 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T21:18:01.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T21:18:01.223 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-23T21:18:01.234 INFO:teuthology.task.internal:Transferring archived files... 2024-03-23T21:18:01.235 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/remote/smithi117 2024-03-23T21:18:01.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T21:18:01.282 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668/remote/smithi204 2024-03-23T21:18:01.282 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T21:18:01.303 INFO:teuthology.task.internal:Removing archive directory... 2024-03-23T21:18:01.304 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T21:18:01.327 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T21:18:01.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-23T21:18:01.359 INFO:teuthology.task.internal:Not uploading archives. 2024-03-23T21:18:01.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-23T21:18:01.369 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-23T21:18:01.369 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T21:18:01.375 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T21:18:01.378 INFO:teuthology.orchestra.run.smithi117.stdout: 658628 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 21:18 /home/ubuntu/cephtest 2024-03-23T21:18:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 21:18 /home/ubuntu/cephtest 2024-03-23T21:18:01.398 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-23T21:18:01.408 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-23T21:18:01.435 INFO:teuthology.nuke:Checking targets against current locks 2024-03-23T21:18:01.463 DEBUG:teuthology.nuke:shortname: smithi117 2024-03-23T21:18:01.463 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T21:18:01.487 DEBUG:teuthology.nuke:shortname: smithi204 2024-03-23T21:18:01.488 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T21:18:01.491 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 20:34:16.383409', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1jcoioqpma6131vrwi23FPocv1RZFh/qwKKJR0unECZSCSz46OXgHJIhVax58gSn+DqTttcf+hwf4JLjyGxfs='} 2024-03-23T21:18:01.511 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-22_13:19:01-rbd-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617668', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 20:34:16.381506', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP3lLbZ167RBEDXN5OmvB8wahaNtp28lVj7Ctgps/XZan0ZhVsbikIRpq37YPGsjrb4IZAdK/KxIOad73zDNz7Q='} 2024-03-23T21:18:01.540 INFO:teuthology.orchestra.console.smithi117:Power off 2024-03-23T21:18:01.540 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T21:18:01.565 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-03-23T21:18:01.565 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T21:18:01.580 INFO:teuthology.orchestra.console.smithi204:Power off 2024-03-23T21:18:01.580 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T21:18:01.589 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-23T21:18:01.602 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-03-23T21:18:01.602 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T21:18:01.625 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-23T21:18:05.590 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T21:18:05.626 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T21:18:05.700 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-23T21:18:05.736 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-23T21:18:09.702 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T21:18:09.737 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T21:18:09.812 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-23T21:18:09.846 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-23T21:18:13.814 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T21:18:13.847 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T21:18:13.924 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-03-23T21:18:13.959 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-03-23T21:18:14.026 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-03-23T21:18:14.060 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-03-23T21:18:14.164 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus} duration: 2056.4460110664368 failure_reason: '"2024-03-23T20:55:05.911428+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: Degraded data redundancy: 1/22 objects degraded (4.545%), 1 pg degraded (PG_DEGRADED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-03-23T21:18:14.164 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T21:18:14.245 INFO:teuthology.run:FAIL